-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Jeff,
> Just discovered the StarCluster system, and enjoyed looking through the
> site. Looks like a very well thought out design. Hope to try it
> sometime in the near future.
Thanks, hope you like it :D
> At Insilicos (www.insilicos.com <http://www.insilicos.com>) we have
> built a similar platform for parallel computing in the AWS cloud. Like
> StarCluster, it uses MPI for internode communication. We’re curious how
> reliable MPI has been for you. Occasionally, we have problems bringing
> up a cluster because mpdboot fails to establish a communications ring.
> It’s pretty clear this is due to higher than usual latency between
> nodes. We take the obvious precaution of making sure all nodes are
> provisioned in the same availability zone, and have even tweaked the
> timeout tolerances inside mpdboot.py, but there are still sporadic bad
> days when the problem occurs.
So first of all StarCluster uses OpenMPI/SSH and not LAM/mpdboot so I
can't really speak to your issues with mpdboot on EC2. I've created a
StarCluster plugin that configures LAM instead of OpenMPI that I'm
planning to add to the StarClusterPlugins project but I've only tested
it on a small number of nodes and for those test runs mpdboot launched
just fine.
Unfortunately I don't personally use MPI all that intensively either. I
occasionally compile and run MPI programs from time to time but nothing
really mission critical. However, I am aware of the high latency in
general between EC2 instances.
I know that Damian Eads (cc'd) uses or has used StarCluster/OpenMPI
quite a bit in the past. Damian, any comments from your experience with
StarCluster/OpenMPI?
Also have you tried the new Compute/GPU instance types? These instance
types require a 'placement group' to be created before an instance is
requested. This group is used to ensure that all instances launched
within the group are closer together and connected via a 10gb network in
order to reduce network latency. I've heard this has significantly
improved MPI performance although I haven't run any benchmarks yet to
investigate...
If you're interested in testing out OpenMPI on these new instance types
I'd suggest installing the latest development version of StarCluster
which has support for launching clusters of the new compute/gpu instance
types. The Cluster Compute/GPU AMI can be found in the output of the
'listpublic' command:
% starcluster listpublic
.....
[4] ami-12b6477b us-east-1
starcluster-base-centos-5.4-x86_64-ebs-hvm-gpu-rc2 (HVM-EBS)
If you do decide to run some benchmarks and wouldn't mind posting your
results to the list that'd be much appreciated.
Hope that helps,
~Justin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla -
http://enigmail.mozdev.org/
iEYEARECAAYFAk15LlkACgkQ4llAkMfDcrmGQwCfQ0bDmAj06u4H0yCFXvTBgv/w
DvQAn2eh7tYZ59aadbTgij+pQkMMxX3W
=6lI0
-----END PGP SIGNATURE-----
Received on Thu Mar 10 2011 - 15:02:50 EST