Re: MIT StarCluster cluster admin web site?

From: Sergio Mafra <no email>
Date: Thu, 31 May 2012 09:27:07 -0300

Hi Rayson,

Thanks for your answer and helpful material. I´ll read it carefully.

All the best,

Sergio

On Wed, May 30, 2012 at 8:12 PM, Rayson Ho <raysonlogin_at_gmail.com> wrote:

> Sergio,
>
> It would be great if you can develep a web interface for StarCluster -
> and I can help you test the code...
>
> For queue status, Grid Engine (or "Open Grid Scheduler" - which is the
> project that maintains the open source grid engine code base) has
> XML-qstat, which was originally developed by Chris from the BioTeam,
> and now it is available at:
>
> http://olesenm.github.com/xml-qstat/
> http://olesenm.github.com/xml-qstat/screenshots.html
>
> There is also a web portal developed by Sun, but I don't think anyone
> is using that one - at least not with any modern versions of Grid
> Engine anyway.
>
> Outside of Grid Engine, there is a lightweight web portal that handles
> job submission, file upload, queue status, etc for OpenPBS (not Torque
> - it was not tested with Torque, AFAIK), and the project is called
> PBSWeb:
>
> http://webdocs.cs.ualberta.ca/~pinchak/PBSWeb/
>
> I talked to Paul Lu (the author) 11 years ago, and he said that PBSWeb
> was developed by him & his grad students. He said that PBSWeb is
> available under the GPL, and he was happy to accept changes from other
> people. According to Paul, internally inside PBSWeb, the web front-end
> invokes an SSH connection to the submission machine for each qsub, and
> this was how he handled user switching (ie. the Apache user has to
> have access to the secret keys of all the users - or else you would
> see root or the Apache User as the submission user for all the jobs).
>
> (BTW, Paul Lu was a student of Songnian Zhou, who was the founder of
> Platform Computing, and Songnian wrote some of the initial Platform
> LSF code when he was at UC Berkeley.)
>
> However, as I am not a web programming guy, I didn't (and still don't)
> know much about web technologies (Yes, I am using Gopher... just
> kidding :-D ). Another thing to look at is the S3 multi-part upload,
> GridFTP, and multi-session SCP upload methods:
>
> http://www.elastician.com/2010/12/s3-multipart-upload-in-boto.html
> http://www.globus.org/toolkit/docs/latest-stable/gridftp/
> http://www.psc.edu/index.php/hpn-ssh
>
> As you might know TCP is too slow to handle large file uploads, so
> those technologies really help. I hope the web portal is flexible
> enough to use at least those technologies by not hard coding to
> specific way of doing the upload.
>
> If you need changes in Grid Engine to integrate with the portal, then
> I can help. Other than that, but then I am not a web guy and thus I am
> not sure what's the best technology to use now in 2012 to tie these
> things together - and may be in this area you know better than I do!
>
> Rayson
>
> ================================
> Open Grid Scheduler / Grid Engine
> http://gridscheduler.sourceforge.net/
>
> Scalable Grid Engine Support Program
> http://www.scalablelogic.com/
>
>
>
> On Wed, May 30, 2012 at 10:17 AM, Sergio Mafra <sergiohmafra_at_gmail.com>
> wrote:
> > Hi fellows,
> >
> > One thing that is missing on a Cluster deployment is, in IMHO, a web site
> > that could serve as a front side to the final users. This site should has
> > some functionality like: Cluster Admin (on/off, cluster health),
> > Upload/Download of files, queue status and others.
> >
> > I have a project to build one like this but I´d like to know first if
> this
> > is available in our community?
> >
> > All the best,
> >
> > Sergio
> >
> >
> > _______________________________________________
> > StarCluster mailing list
> > StarCluster_at_mit.edu
> > http://mailman.mit.edu/mailman/listinfo/starcluster
> >
>
>
>
> --
> ==================================================
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
>
Received on Thu May 31 2012 - 08:27:09 EDT