Re: [Starcluster] Multiple MPI jobs on SunGrid Engine with StarCluster

From: Damian Eads <no email>
Date: Mon, 21 Jun 2010 23:39:41 -0700

Hi,

It's working perfectly now. Just queued up 110 20-core MPI jobs and
all 8 cores are being used on every node.

Thanks so much for your clarifications and explanations. They helped
me get on the right track.

Cheers,

Damian



On Mon, Jun 21, 2010 at 8:33 PM, Justin Riley <jtriley_at_mit.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Damian,
>
> Could you try removing the -hostfile/-wd/-n options to mpirun and see if
> that works? Also the "./tesla-train --config $2" line is on the same
> line as the mpirun in your testla-train.sh right?
>
> ~Justin
>
> On 06/21/2010 08:14 PM, Damian Eads wrote:
>> Hi Justin,
>>
>> Please see the attached image. It shows the output of top when I try
>> to start a 20 process OpenMPI job using qsub. Notice that it doesn't
>> spread the processes across the cluster.
>>
>> root_at_domU-12-31-39-0E-29-81:/data/tesla# start-large.sh birds-1.cfg
>>
>> root_at_domU-12-31-39-0E-29-81:/data/tesla# cat
>> ../teslaconf/scripts/start-large.sh
>> #!/bin/bash
>>
>> NP=20
>>
>>
>> for i in $*
>> do
>>   JOBNAME=`echo $i | sed 's/\.cfg//g' | sed 's/^.*\///g'`
>>   OUTFILE=/tmp/az-${JOBNAME}.out
>>   ERRFILE=/tmp/az-${JOBNAME}.err
>>   export PATH=/data/teslaconf/scripts:/data/tesla:$PATH
>>   CMD="qsub -N ${JOBNAME} -pe orte ${NP} -o ${OUTFILE} -e ${ERRFILE}
>> -wd /data/tesla -V ../teslaconf/scripts/tesla-train.sh ${NP} $i"
>>   echo Running $CMD
>>   $CMD
>> done
>>
>> root_at_domU-12-31-39-0E-29-81:/data/tesla# cat
>> ../teslaconf/scripts/tesla-train.sh
>> #!/bin/bash
>>
>> PYTHON_VERSION=python2.6
>> PYTHONPATH=/data/prefix/python2.6/site-packages
>>
>> mpirun -hostfile /data/teslaconf/machines -wd /data/tesla \
>>        -n $1 -x PYTHONPATH=/data/prefix/lib/python2.6/site-packages
>> ./tesla-train --config $2
>>
>> Ideas?
>>
>> Thanks,
>>
>> Damian
>>
>> On Mon, Jun 21, 2010 at 10:02 AM, Damian Eads <eads_at_soe.ucsc.edu> wrote:
>>> Hi Justin,
>>>
>>> Thank you for your explanation on MPI/SGE integration. A few quick questions:
>>>
>>> On Mon, Jun 21, 2010 at 9:46 AM, Justin Riley <jtriley_at_mit.edu> wrote:
>>>> So the above explains the parallel environment setup within SGE. It
>>>> turns out that if you're using a parallel environment with OpenMPI, you
>>>> do not have to specify --byslot/--bynodes/-np/-host/etc options to
>>>> mpirun given that SGE will handle the round_robin/fill_up modes for you
>>>> and automatically assign hosts and number of processors to be used by
>>>> OpenMPI.
>>>>
>>>> So, for your use case I would change the commands as follows:
>>>> - -----------------------------------------------------------------
>>>>    qsub -pe orte 24 ./myjobscript.sh experiment-1
>>>>    qsub -pe orte 24 ./myjobscript.sh experiment-2
>>>>    qsub -pe orte 24 ./myjobscript.sh experiment-3
>>>>    ...
>>>>    qsub -pe orte 24 ./myjobscript.sh experiment-100
>>>>
>>>> where ./myjobscript.sh calls mpirun as follows
>>>>
>>>>    mpirun -x PYTHONPATH=/data/prefix/lib/python2.6/site-packages
>>>>                 -wd /data/experiments ./myprogram $1
>>>
>>> So I don't need to specify the number of slots to use in mpirun? The
>>> Sun GridEngine will somehow pass this information to mpirun? Is the
>>> mpirun argument -n 1 by default?
>>>
>>>> - -----------------------------------------------------------------
>>>> NOTE: You can also pass -wd to the qsub command instead of mpirun and
>>>> along the same lines I believe you can pass -v option to qsub rather
>>>> than -x to mpirun. Neither of these should make a difference, just
>>>> shifts where the -x/-wd concern is (from MPI to SGE).
>>>
>>> Good idea. This will clean things up a bit. Thanks for suggesting it.
>>>
>>>> I will add a section to the docs about using SGE/OpenMPI integration on
>>>> StarCluster based on this email.
>>>>
>>>>> Perhaps if carefully
>>>>> used, this will ensure that there is a root MPI process running on the
>>>>> master node for every MPI job that's simultaneously running.
>>>>
>>>> Is this a requirement for you to have a root MPI process on the master
>>>> node for every MPI job? If you're worried about oversubscribing the
>>>> master node with MPI processes, then this SGE/OpenMPI integration should
>>>> relieve those concerns. If not, what's the reason for needing a 'root
>>>> MPI process' running on the master node for every MPI job?
>>>
>>> It's not a requirement but reflects some ignorance on my part. <aybe
>>> I'm confused about why the first node is called master. I was assuming
>>> it had that name because it was performing some kind of special
>>> coordination.
>>>
>>> Do I still need to provide the -hostfile option? Or is this automatic now?
>>>
>>> Thanks a lot!
>>>
>>> Damian
>>>
>>> -----------------------------------------------------
>>> Damian Eads                           Ph.D. Candidate
>>> University of California             Computer Science
>>> 1156 High Street         Machine Learning Lab, E2-489
>>> Santa Cruz, CA 95064    http://www.soe.ucsc.edu/~eads
>>>
>>
>>
>>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.14 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkwgLwgACgkQ4llAkMfDcrkMjgCgkQatuSAT805MA01R97R8ezKY
> GNUAniGxxptbNo8nRavZwEOAx/lfI6oK
> =CgAJ
> -----END PGP SIGNATURE-----
>

-- 
-----------------------------------------------------
Damian Eads                           Ph.D. Candidate
University of California             Computer Science
1156 High Street         Machine Learning Lab, E2-489
Santa Cruz, CA 95064    http://www.soe.ucsc.edu/~eads

Received on Tue Jun 22 2010 - 02:39:43 EDT