Re: Local / cluster folder bi-directional synchronization

From: MacMullan, Hugh <no email>
Date: Sat, 19 Jan 2013 23:31:39 +0000

Hi Olivier:

I've had good success with Dropbox, either using a custom AMI with the package already installed, or having it available in a StarCluster mounted volume, and then doing a 'dropbox start' as the first line of any script, or the first script run on cluster start. Trick is to use an ElasticIP for master, and you also have to register once ( per IP/user combination) ... Assuming you're also using the same homedir mounted from an EBS volume.

Lot of requirements, so would be a tricky plugin to write. But I'm generally working with custom AMIs anyway so it's not a stretch for our usage.

Good luck if you proceed with unison! I'll play with it if you come up with something, for sure.

-Hugh

On Jan 19, 2013, at 6:04 PM, "Olivier Grisel" <olivier.grisel_at_ensta.org> wrote:

> Hi all,
>
> I would like to be able to setup a bi-directional link between the my
> home folder in the EC2 cluster and some local folder on my laptop
> (e.g. without my working code and ipython notebook under git version
> control).
>
> I currently toying with unison [1] and I think I will be able to turn
> that into a starcluster plugin.
>
> [1] http://www.cis.upenn.edu/~bcpierce/unison/
>
> I think it should also be quite easy to write a plugin to setup a
> reverse ssh tunnel and then use fuse's sshfs command to mount my
> laptop folder onto some mount point folder on the cluster (assuming
> ssh server is running on my laptop). However the link my break and I
> don't want to interrupt a long running computation on the cluster when
> I put my laptop to sleep, so the unison approach sounds more
> promising.
>
> Before going further, has anybody else worked on similar tools? Any
> tips or resources to share?
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Sat Jan 19 2013 - 18:31:43 EST