Re: New Grid Engine Hadoop Integration HOWTO

From: Paul McDonagh <no email>
Date: Fri, 1 Jun 2012 15:31:53 -0400

Thanks Rayson,

This and the previous email are a couple of really good suggestions. I'll try 'em out and see what happens.

Best,
Paul.

On Jun 1, 2012, at 14:52, Rayson Ho wrote:

> If you are running Hadoop on StarCluster, you may also be interested
> in this new method contributed by Prakashan Korambath of UCLA.
>
> http://gridscheduler.sourceforge.net/howto/GridEngineHadoop.html
>
> The difference between the original SGE 6.2u5 method vs the new one is
> that with Prakashan's approach, Grid Engine is used for resource
> allocation, and the Hadoop job scheduler/Job Tracker is used to handle
> all the MapReduce operations. A Hadoop cluster is created on demand
> with Prakashan's approach, but in the original SGE 6.2u5 method Grid
> Engine replaces the Hadoop job scheduler.
>
> As standard Grid Engine PEs are used in this new approach, one can
> call "qrsh -inherit" and use Grid Engine's method to start Hadoop
> services on remote nodes, and thus get full job control, job
> accounting, and cleanup at terminate benefits like any other tight PE
> jobs!
>
> Rayson
>
> ================================
> Open Grid Scheduler / Grid Engine
> http://gridscheduler.sourceforge.net/
>
> Scalable Grid Engine Support Program
> http://www.scalablelogic.com/
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Fri Jun 01 2012 - 15:31:57 EDT