HDF5

HDF5 is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids.

Availability and Restrictions

HDF5 is available on Glenn, Oakley and Ruby Clusters. The versions currently available at OSC are:

Version Glenn Oakley Ruby notes
1.6.4 X(P)      
1.6.5 X*      
1.8.7 X      
1.8.8   X    
1.8.12     X(GI) Default version on Ruby prior to 09/14/2015
1.8.13   X(GI)

X(GI)

Default version on Oakley prior to 09/14/2015
1.8.15   X* X(PI)*  
  • X(P): pgi only
  • X(PI): available with pgi and intel
  • X(GI): available with gnu and intel
  • X; available with pgi, gnu, and intel
* : Current Default Version

You can use module spider hdf5  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

HDF5 is available without restriction to all OSC users.

Usage

Usage on Glenn

Set-up on Glenn

Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library on Glenn, run the following command: module load hdf5. To load a particular version, use  module load name. For example, use  module load hdf5-1.6.5-intel  to load HDF5 version 1.6.5 with the Intel Compiler on Glenn. You can use module avail hdf5  to view available modules on Glenn.

Building With HDF5

The HDF5 library provides the following variables for use at build time:

Variable Use
$HDF5_C_INCLUDE Use during your compilation step for C programs
$HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
$HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
$HDF5_C_LIBS Use during your linking step programs
$HDF5_F90_LIBS Use during your linking step for FORTRAN programs

For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

icc -c $HDF5_C_INCLUDE myprog.c
icc -o myprog myprog.o $HDF5_C_LIBS
ifort -c $HDF5_F90_INCLUDE myprog.f90
ifort -o myprog myprog.o $HDF5_F90_LIBS

Batch Usage on Glenn

When you log into glenn.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Non-interactive Batch Job (Serial Run)
A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
#PBS -N AppNameJob
#PBS -l nodes=1:ppn=8
module load hdf5
cd $PBS_O_WORKDIR
cp foo.dat $TMPDIR
cd $TMPDIR
appname
cp foo_out.h5 $PBS_O_WORKDIR

Usage on Oakley

Set-up on Oakley

Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library on Oakley, run the following command: module load hdf5. To load a particular version, use  module load hdf5/version. For example, use  module load hdf5/1.8.15 to load HDF5 version 1.8.15 on Oakley. You can use module spider hdf5  to view available modules on Oakley.

Building With HDF5

The HDF5 library provides the following variables for use at build time:

Variable Use
$HDF5_C_INCLUDE Use during your compilation step for C programs
$HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
$HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
$HDF5_C_LIBS Use during your linking step programs
$HDF5_F90_LIBS Use during your linking step for FORTRAN programs

For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

icc -c $HDF5_C_INCLUDE myprog.c
icc -o myprog myprog.o $HDF5_C_LIBS
ifort -c $HDF5_F90_INCLUDE myprog.f90
ifort -o myprog myprog.o $HDF5_F90_LIBS

Batch Usage on Oakley

When you log into oakley.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info.  Batch jobs run on the compute nodes of the system and not on the login node. It is desirable for big problems since more resources can be used.

Non-interactive Batch Job (Serial Run)
A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
#PBS -N AppNameJob
#PBS -l nodes=1:ppn=12
module load hdf5
cd $PBS_O_WORKDIR
cp foo.dat $TMPDIR
cd $TMPDIR
appname
cp foo_out.h5 $PBS_O_WORKDIR

Usage on Ruby

Set-up on Ruby

Initalizing the system for use of the HDF5 library is dependent on the system you are using and the compiler you are using. To load the default HDF5 library on Ruby, run the following command: module load hdf5. To load a particular version, use  module load hdf5/version. For example, use  module load hdf5/1.8.15 to load HDF5 version 1.8.15 on Ruby. You can use module spider hdf5  to view available modules on Ruby.

Building With HDF5

The HDF5 library provides the following variables for use at build time:

Variable Use
$HDF5_C_INCLUDE Use during your compilation step for C programs
$HDF5_CPP_INCLUDE Use during your compilation step for C++ programs (serial version only)
$HDF5_F90_INCLUDE Use during your compilation step for FORTRAN programs
$HDF5_C_LIBS Use during your linking step programs
$HDF5_F90_LIBS Use during your linking step for FORTRAN programs

For example, to build the code myprog.c or myprog.f90 with the hdf5 library you would use:

icc -c $HDF5_C_INCLUDE myprog.c
icc -o myprog myprog.o $HDF5_C_LIBS
ifort -c $HDF5_F90_INCLUDE myprog.f90
ifort -o myprog myprog.o $HDF5_F90_LIBS

Batch Usage on Ruby

When you log into ruby.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations and Batch Limit Rules for more info. 

Non-interactive Batch Job (Serial Run)
A batch script can be created and submitted for a serial or parallel run. You can create the batch script using any text editor you like in a working directory on the system of your choice. Below is the example batch script that executes a program built with the HDF5 library:
#PBS -N AppNameJob
#PBS -l nodes=1:ppn=20
module load hdf5
cd $PBS_O_WORKDIR
cp foo.dat $TMPDIR
cd $TMPDIR
appname
cp foo_out.h5 $PBS_O_WORKDIR

Further Reading

See Also

Supercomputer: 
Service: