News
Abstract Generic Data Graph Image for Evaluating Computational Performance in Weather Model Testing

Evaluating Computational Performance in Weather Model Testing

Author: Zachary Shrader
EPIC Team: Code Management Systems Engineer & Code Manager

Wall Clock:

The following diagrams show the wall clock times of the UFS Weather Model regression tests run by the Code Management team against prospective code changes in active GitHub pull requests (PRs). The blue represents regression tests that utilize Intel compilers to build the physics suites for the tests; conversely the purple stacked columns represent GNU compilers. Every machine uses some version of Intel, while only Hercules and Hera have a GNU compiler option. The plots showcase the aggregate results of runs over the previous year by the Code Management team and provide the opportunity to compare machines and compiler versions. Derecho and Hercules appear to be the fastest machines, in terms of wall time per test measured in seconds, with no test running for longer than 1,400 and 1,300 seconds, respectively. Hera holds the longest run times at a high mark of 2,400 seconds combined for both Intel and GNU versions of some of the coupled tests. This could be due to Hera running somewhat older compiler versions: intel/2021.5.1 and gnu/9.2.0 for Hera and intel/2021.9.0 and gnu/12.2.0 for Derecho. Jet is also quite slow, reaching 2,000 seconds for some coupled tests. This is likely due to Jet being a smaller machine with fewer resources and less computational power.

Visualization Graph of Regression Test Wall Time performances on Derecho HPC. (Intel compilers only)
Regression Test Wall Time performances on Derecho HPC. (Intel compilers only)
Visualization Graph of Regression Test Wall Time performances on Gaea-C5 HPC. (Intel compilers only)
Regression Test Wall Time performances on Gaea-C5 HPC. (Intel compilers only)
Visualization Graph of Regression Test Wall Time performances on Hera HPC. (Intel compilers in blue and GNU compilers in purple)
Regression Test Wall Time performances on Hera HPC. (Intel compilers in blue and GNU compilers in purple)
Visualization Graph of Regression Test Wall Time performances on Hercules HPC. (Intel compilers in blue and GNU compilers in purple)
Regression Test Wall Time performances on Hercules HPC. (Intel compilers in blue and GNU compilers in purple)
Visualization Graph of Regression Test Wall Time performances on Jet HPC. (Intel compilers only)
Regression Test Wall Time performances on Jet HPC. (Intel compilers only)

Here we can see the number of regression tests per compiler on each machine. Most of the regression tests are Intel-based. The operational requirement tests, represented in green, are run almost exclusively with GNU compilers. While GNU compilers are generally slower, as compared to Intel, they are also less tolerant in the detection of bugs. This makes them a good choice for smoke testing PR code changes.

Visualization Graph of Number of Regression Tests cases per HPC compiler environment in red. Operational Requirement tests in green, only run on Hera.
Number of Regression Tests cases per HPC compiler environment in red. Operational Requirement tests in green, only run on Hera.
Visualization Graph of Number of Tests cases performed on each machine, during Code Management, separated by compiler type. Intel in blue and GNU in purple.
Number of Tests cases performed on each machine, during Code Management, separated by compiler type. Intel in blue and GNU in purple.