StarCluster - Mailing List Archive

Re: Possible to bring in data stored on S3?

From: Jennifer Staab <no email>
Date: Mon, 06 Oct 2014 21:49:54 -0400

I agree with Nik, it seems reasonably fast in my experience. Note that
S3 doesn't have a file system, it's just an indexed object. So you can
write your object once and read it many times, but there isn't any
updating, file locking, etc. I typically use S3 for storage and
archiving or to provide a static data reference for things that aren't
going to change, but need to be widely available. I know those that use
S3 for static webpages, or for storing logs, etc. If your data is going
to change often, you might want to do things a different way.

Good Luck.


On 10/6/14 9:19 PM, John Readey wrote:
> Is anyone pulling in data on demand using the S3 API's? I'm curious
> how the performance compares to using NFS.
> John
> From: Jennifer Staab < <>>
> Date: Monday, October 6, 2014 at 6:08 PM
> To: greg < <>>,
> " <>"
> < <>>
> Subject: Re: [StarCluster] Possible to bring in data stored on S3?
> Besides the others mentioned, AWS has their own "command line
> interface" CLI see here
> <>
> for installation instructions and here
> <> for usage. AWS CLI
> provides tools for accessing S3. If you don't want to use AWS CLI,
> there are many SDKs available depending on programming language or
> platform see here <>.
> Good Luck.
> Jennifer
> On 10/6/14 3:29 PM, greg wrote:
>> Hi all,
>> Is it possible to bring data into my cluster from Amazon S3? I didn't
>> see it in the manual.
>> Thanks for reading!
>> Greg
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster_at_mit.edu
Received on Mon Oct 06 2014 - 21:49:58 EDT
This archive was generated by hypermail 2.3.0.


Sort all by: