Re: Terminating a cluster does not remove its security group (UNCLASSIFIED)

From: Oppe, Thomas C ERDC-RDE-ITL-MS Contractor <no email>
Date: Thu, 20 Dec 2012 13:19:08 +0000

Classification: UNCLASSIFIED
Caveats: FOUO

Dustin,

I'm running a HYCOM job now using the /mnt disk on the master node. The problem is that big files are being written. One of the files is 57GB, and 24 other files of size 4GB are also written. The job is running more slowly than when run on an EBS volume, but that may be because the MPI executable is on an EBS volume and is being run from there.

I gave up trying to login to each node of the cluster using "starcluster sshnode cluster <node>", mainly because I was getting ssh errors that were preventing me from logging in to several nodes. The error message was something like "you may be in a man-in-the-middle attack" or something like that.

My technique is very kludgy, but this is what I did. I wrote a bash script to copy four HYCOM input files that each MPI process needs to be able to read to the /mnt disk. /sharedWork is the name of my standard EBS volume.

root_at_master > more copyit
#!/bin/bash
cd /sharedWork/ABTP/ded/hycom/input
/bin/cp blkdat.input limits ports.input /mnt
/bin/cp patch.input_00336 /mnt/patch.input
ls /mnt
exit

Then I wrote a small Fortran MPI program that executes this script with a system call. Since there are 16 MPI processes to an instance (cc2.8xlarge), I only use one MPI process on each node to do the copying.

root_at_master > more copy.f
      program copy
      include 'mpif.h'
      call MPI_Init (ierror)
      call MPI_Comm_rank (MPI_COMM_WORLD, myid, ier)
      if (mod(myid, 16) .eq. 0) then
         call system ('/sharedWork/copyit')
      endif
      call MPI_Finalize (ierror)
      stop
      end

I compiled "copy.f" into an executable called "copy" using "mpiifort". Then in my HYCOM batch script, I run this small MPI job before running the real job:

mpiexec -n 336 /sharedWork/copy

Unfortunately, I think I made an error in that the HYCOM executable is being run from /sharedWork, the EBS volume.

mpiexec -n 336 /sharedWork/hycom

The job is running slowly, more slowly than if all files were on /sharedWork. For the next run, I think I will modify the "copyit" script to copy the HYCOM executable to /mnt also for each instance and change the mpiexec line to:

mpiexec -n 336 /mnt/hycom

HYCOM needs a lot of input files, and they are all available on /mnt corresponding to the master node (rank 0). However, most of the input files are read only by rank 0 and only four input files need to be read by all MPI processes.

I'm surprised this works at all. It is mind-bending for me. If you have a better way for running on /mnt, let me know your suggestion.

Above all, thank you very much. I think once the executable is placed on each /mnt disk for each node, maybe the run will go much faster.

Tom Oppe

-----Original Message-----
From: Dustin Machi [mailto:dmachi_at_vbi.vt.edu]
Sent: Thursday, December 20, 2012 5:53 AM
To: Oppe, Thomas C ERDC-RDE-ITL-MS Contractor
Cc: Paolo Di Tommaso; starcluster_at_mit.edu
Subject: Re: [StarCluster] Terminating a cluster does not remove its security group (UNCLASSIFIED)

> I'm looking at using the /mnt local disk of the "master" node and
> copying input files to each node's /mnt disk for those files that each
> MPI process needs to read. It's very cumbersome. I have to login to
> each node.

It is a little cumbersome to do so, but you shouldn't have to login to each node. Just submit/run an SGE job that copies/rsyncs the NFS volume mounted on each machine to /mnt/. Is your problem a read problem or a write problem?

Dustin

Classification: UNCLASSIFIED
Caveats: FOUO
Received on Thu Dec 20 2012 - 08:19:14 EST