HDF5 Users

From: John Readey <no email>
Date: Tue, 16 Sep 2014 18:50:16 +0000

Hello,
  I just came across the StarCluster project and was wondering how popular the HDF5 file format is with StarCluster users. One challenge with using HDF5 in a cluster is that HDF5 over NFS is not supported. Currently parallel usage of HDF5 now is restricted to MPI-I/O with parallel filesystems such as Lustre or GPFS.
  At the HDFGroup we've been proto-typing a REST-based service implementation of HDF5 - see http://www.hdfgroup.org/pubs/papers/RESTful_HDF5.pdf for a description of the API. I can imagine this service being a good solution for cluster based data I/o. In this scenario each node in the cluster would read/write data via http requests (rather than using NFS). Performance is one aspect that would be critical in a cluster environment. It's unknown at this point how much overhead the http request/response processing would add.
  I touched on this idea in my presentation at this summer's HDF Workshop. You can see the slides here: http://www.slideshare.net/HDFEOS/jxr?ref=http://hdfeos.org/workshops/ws17/slideshare.php. From the architecture slide you can see that clients can access the REST API directly or use the HDF5 library (for C/Fortran applications). The library backend will convert the calls to http requests.
  If anyone is interested in this approach it would be great to discuss. Please reply to this thread or contact me directly.

John Readey
HDF Group
http://www.hdfgroup.org/
Jreadey AT hdfgroup.org
Received on Tue Sep 16 2014 - 14:50:20 EDT