Re: Terminating a cluster does not remove its security group (UNCLASSIFIED)
This archive was generated by
I'm running a HYCOM job now using the /mnt disk on the master node. The problem is that big files are being written. One of the files is 57GB, and 24 other files of size 4GB are also written. The job is running more slowly than when run on an EBS volume, but that may be because the MPI executable is on an EBS volume and is being run from there.
I gave up trying to login to each node of the cluster using "starcluster sshnode cluster <node>", mainly because I was getting ssh errors that were preventing me from logging in to several nodes. The error message was something like "you may be in a man-in-the-middle attack" or something like that.
My technique is very kludgy, but this is what I did. I wrote a bash script to copy four HYCOM input files that each MPI process needs to be able to read to the /mnt disk. /sharedWork is the name of my standard EBS volume.
root_at_master > more copyit
/bin/cp blkdat.input limits ports.input /mnt
/bin/cp patch.input_00336 /mnt/patch.input
Then I wrote a small Fortran MPI program that executes this script with a system call. Since there are 16 MPI processes to an instance (cc2.8xlarge), I only use one MPI process on each node to do the copying.
root_at_master > more copy.f
call MPI_Init (ierror)
call MPI_Comm_rank (MPI_COMM_WORLD, myid, ier)
if (mod(myid, 16) .eq. 0) then
call system ('/sharedWork/copyit')
call MPI_Finalize (ierror)
I compiled "copy.f" into an executable called "copy" using "mpiifort". Then in my HYCOM batch script, I run this small MPI job before running the real job:
mpiexec -n 336 /sharedWork/copy
Unfortunately, I think I made an error in that the HYCOM executable is being run from /sharedWork, the EBS volume.
mpiexec -n 336 /sharedWork/hycom
The job is running slowly, more slowly than if all files were on /sharedWork. For the next run, I think I will modify the "copyit" script to copy the HYCOM executable to /mnt also for each instance and change the mpiexec line to:
mpiexec -n 336 /mnt/hycom
HYCOM needs a lot of input files, and they are all available on /mnt corresponding to the master node (rank 0). However, most of the input files are read only by rank 0 and only four input files need to be read by all MPI processes.
I'm surprised this works at all. It is mind-bending for me. If you have a better way for running on /mnt, let me know your suggestion.
Above all, thank you very much. I think once the executable is placed on each /mnt disk for each node, maybe the run will go much faster.
From: Dustin Machi [mailto:dmachi_at_vbi.vt.edu]
Sent: Thursday, December 20, 2012 5:53 AM
To: Oppe, Thomas C ERDC-RDE-ITL-MS Contractor
Cc: Paolo Di Tommaso; starcluster_at_mit.edu
Subject: Re: [StarCluster] Terminating a cluster does not remove its security group (UNCLASSIFIED)
> I'm looking at using the /mnt local disk of the "master" node and
> copying input files to each node's /mnt disk for those files that each
> MPI process needs to read. It's very cumbersome. I have to login to
> each node.
It is a little cumbersome to do so, but you shouldn't have to login to each node. Just submit/run an SGE job that copies/rsyncs the NFS volume mounted on each machine to /mnt/. Is your problem a read problem or a write problem?
Received on Thu Dec 20 2012 - 08:19:14 EST