An Introduction to the SPEC High Performance Group and Their Benchmark Suites

Robert Henschel* and Huian Li
Pervasive Technology Institute, Indiana University
Bloomington, Indiana 47408, USA,

* Presenting author

The High Performance Group(HPG) of the Standard Performance Evaluation Corporation(SPEC) is a forum for discussing and developing benchmark methodologies for High Performance Computing(HPC) systems. At the same time, the group produces production quality benchmark suites like SPEC MPI2007, SPEC OMP2012 and SPEC ACCEL. The Scientific Applications and Performance Tuning group of the Pervasive Technology Institute at Indiana University(IU) has used these benchmark suites for years to conduct research in HPC systems and facilitate procurement, testing, tuning and monitoring of HPC systems. In this presentation we provide an overview of the SPEC High Performance Group and their benchmark suites by describing our methodology of using the benchmarks to access compiler performance, tune system parameters, evaluate scalability, and monitor power consumption.

The High Performance Group of SPEC has been developing benchmark suites for over twenty years and is comprised of HPC vendors and research institutions from all over the world. The group collaboratively develops application benchmark suites that target the available parallelization paradigms and allow users to compare overall system performance. SPEC HPG benchmark suites are based on real world parallel scientific applications and go beyond synthetic kernel benchmarks to allow users to better understand real world system performance. An important part of the work is peer reviewing the results and publishing them in a repository on the SPEC web page. This curated result repository is freely available and can be used to model and estimate performance of a wide range of HPC systems.

At IU, we use SPEC HPG benchmark suites for research and for day-to-day operations of an HPC center. For example, we include SPEC HPG benchmark suites in the HPC procurement process. Together with real world applications, they are part of the Request for Proposal(RFP) and the acceptance test criteria. Since SPEC HPG benchmark suites cover a wide field of parallelism, such as accelerator performance (SPEC ACCEL), single node performance (SPEC OMP2012) and interconnect performance (SPEC MPI2007), they enable us to predict and model the performance of a new system and evaluate vendor responses to the RFP.

We have also used the benchmark suites for research in HPC system design. We evaluated the performance of parallel scientific application in a Virtual Machine (VM) environment while at the same time measuring power consumption. This has provided us with new insights into the power efficiency of VM environments under real world load.

The SPEC ACCEL benchmark suite contains OpenCL and OpenACC components. Since Indiana University is a member of the OpenACC Standards Group, we are using the SPEC ACCEL benchmark suite to track the performance of OpenACC compilers over time. Compared to compilers for other parallelization paradigms, OpenACC compilers are quite new and users can expect rapid performance improvements as the compilers mature. In the presentation, we will show a performance comparison of the PGI OpenACC and the Cray OpenACC compiler and discuss how they have evolved over time. In addition, we will discuss efforts to add an OpenMP 4.0 component to the accelerator benchmark.

In the presentation, concrete benchmark numbers on IBM, Lenovo and other HPC systems will be shown.

[download slides]