Challenges of Computing with FLASH, a Highly Capable Multiphysics Multiscale AMR Code, on Leadership Class Machines

Anshu Dubeya,b, Christopher Daleyb, George Jordanb, Paul Richb, Donald Q. Lambb, Rob Lathamc, Katherine Rileyc, Dean Townsleyd, Klaus Weideb

bASC/Flash Center, The University of Chicago, 5640 S. Ellis Ave, Chicago, IL 60637
cArgonne National Laboratory, 9700 S. Cass Ave, Argonne, IL, 60439
dDepartment of Physics and Astronomy, University of Alabama, Box 870324, Tuscaloosa, AL 35487

FLASH is a highly capable, fully modular, professionally managed code with a wide user base. FLASH consists of inter-operable modules that can be combined to generate dirent applications such as simulations of novae, supernovae, X-Ray bursts, galaxy clusters, weakly compressible turbulence, and many other problems in astrophysics and other elds. FLASH is regularly used on largest available HPC platforms. With each new platform we encounter new set of challenges because the multiscale multiphysics nature of FLASH simulations and large volumes of output data generated exercise the machine's hardware and system software. Our most recent experience has been with the IBM BG/P Intrepid at Argonne National Laboratory, where we have used more than 120 million CPU hours under the Early Science and Incite programs. In addition to the usual new platform challenges, the high degree of concurrency in this platform also exposed the limits within some of FLASH's parallel algorithms and its I/O organization. In this presentation we will discuss our experiences and limitations found, and their resolution in both FLASH and Intrepid.

ScicomP logo