Parallel I/O using standard data formats in climate and NWP

Deike Kleberg and Luis Kornblueh, Max Planck Institute for Meteorology, Hamburg, Germany

Climate and NWP are two subjects having standardized data formats dating back to the early 1980s. In terms of size of the resulting data WMO GRIB is the most compact one. Additional compression is possible and the overall size of the data is around 15% of the according original 8 byte double floating point numbers. Unfortunately the required chunking of data is transposed to the models domain decomposition. Furthermore is it not possible to predict the size of each single record. IBM MPI/IO is having tremendous problems with this I/O scenario. Because of this we have been exploring different strategies to replace MPI I/O on top of GPFS.

ScicomP logo