Continuous Benchmarking of HPC Simulations

Opportunities and Challenges in Implementing Continuous Benchmarking for HPC Simulations

Ensuring performance consistency and early regression detection is critical in high-performance computing (HPC) operations. Traditional benchmarking methods rely on manual execution, leading to inconsistencies and delayed issue detection. During the JUREAP program, we integrated Continuous Benchmarking (CB) using exacb to standardize performance evaluation across 50 applications. This automation improved reproducibility, streamlined reporting, and enabled early detection of system anomalies, such as faulty Slurm updates and workflow execution issues on the JEDI machine. Even without a fully operational exascale supercomputer, exacb facilitated systematic performance comparisons, providing valuable insights into application scalability.

Beyond JUREAP, Cx enhances research software development and HPC system management. Our framework simplifies benchmarking, ensuring efficient performance tracking and optimization at scale—key for the upcoming JUPITER exascale supercomputer. Automating benchmarking reduces manual overhead, improves system stability, and aids in troubleshooting by providing structured performance insights. In this talk, we share our experience implementing CB in JUREAP, key findings from benchmarking 50 applications, and the broader impact of CI/CD/CB on research software, system administration, and future exascale computing.