LLNL-led SCREAM team clinches inaugural Gordon Bell Prize for climate modeling

A Lawrence Livermore National Laboratory (LLNL)-led effort that performed an unprecedented global climate model simulation on the world’s first exascale supercomputer has won the first-ever Association for Computing Machinery (ACM) Gordon Bell Prize for Climate Modelling, ACM officials announced Thursday.

The Simple Cloud Resolving E3SM Atmosphere Model (SCREAM) team, led by LLNL staff scientist Peter Caldwell and including researchers from Sandia National Laboratories (Sandia) and six other Department of Energy (DOE) national laboratories, took home the prestigious prze at the International Conference for High Performance Computing, Networking, Storage and Analysis (SC23) in Denver. The prize — the highest honor in high performance computing for the climate modeling category — was established to recognize “innovative parallel computing contributions toward solving the global climate crisis,” according to ACM. The award will be given out annually for the next 10 years to recognize the contributions of climate scientists and software engineers in addressing climate change.

“The Gordon Bell Prize has been an inspiration to DOE’s climate scientists for many years (and doubly so for this new climate award),” Caldwell said. “In that sense, Gordon Bell has accomplished his goal of pushing climate science forward.”

The SCREAM team’s winning Gordon Bell submission, led by Energy Exascale Earth System Model (E3SM) chief computational scientist Mark Taylor of Sandia, presents the efficient and performance-portable implementation of SCREAM. SCREAM is the first global cloud-resolving model to run on an exascale supercomputer and the first model to break the one-simulated-year-per-day barrier for realistic cloud-resolving simulations at less than five kilometers of horizontal resolution: achieving 1.26 simulated years per day using 32,768 graphics-processing units (GPUs) on 8,192 nodes of Oak Ridge National Laboratory’s Frontier supercomputer. It also is the first to run on both AMD and NVIDIA GPUs, as well as conventional CPUs. This benchmark time is about 50% faster than the initial prize submission, thanks mainly to improved compile-time settings, researchers said. The paper also discusses the benefits of running the model on an exascale system and how it can be used to improve scientific understanding of climate change and weather patterns.

“This award is a testament to teamwork,” Caldwell said. “It took us five years and the equivalent of six full-time employees spread across about 15 staff members to get where we are. We come from an interesting mix of backgrounds: half of the people are atmospheric physicists and the other half are computer scientists or applied mathematicians. One of the major accomplishments of this effort was just learning how to work together across disciplines.”

The Gordon Bell-winning paper includes researchers and computational scientists at LLNL, Sandia and the Argonne, Brookhaven, Los Alamos, Lawrence Berkeley, Oak Ridge and Pacific Northwest national laboratories.

Caldwell singled out Sandia’s Andy Salinger for coming up with SCREAM’s performance-portability strategy, and LLNL scientist Dave Bader, leader of the E3SM project that SCREAM is part of, for his “unwavering support,” as well as DOE’s Office of Biological and Environmental Research for sponsoring the work. Other LLNL staff named in the Gordon Bell entry include scientists Aaron Donahue, Chris Terai and Renata McCoy.

As a full-featured atmospheric general-circulation model developed for very fine-resolution simulations on exascale machines, SCREAM incorporates state-of-the-art parameterizations for fluid dynamics, microphysics, moist turbulence and radiation. SCREAM can explicitly resolve large convective circulations and other important atmospheric phenomena, thereby avoiding critical sources of uncertainty in traditional climate models. By running SCREAM at exascale, E3SM can provide more trustworthy and higher-fidelity predictions of future climate for longer durations than were previously possible, researchers said.

Sandia’s Luca Bertagna, who presented the paper at the SC23 conference, noted that by rewriting SCREAM in Kokkos, a C++ library for on-node parallelism, the team was able to run efficiently across the spectrum of computing architectures, including heterogenous supercomputers using either AMD or NVIDIA GPUs (with support for Intel GPUs on the way). 

Bertagna emphasized that “what we did for Gordon Bell was not just for Gordon Bell, we actually need this stuff… All of these simulations we run we use the same preparation that we use for the Gordon Bell submission, so we’re using that configuration to answer actual climate-modeling questions.” In particular, SCREAM has already completed a set of 40-day simulations for all four seasons and has almost finished a pair of 13-month simulations aimed at estimating climate sensitivity.  

The team is currently gearing up to launch a pair of multi-decadal simulations used to capture the statistics of rare high-impact events.

SCREAM researchers also said the work is a significant milestone, establishing E3SM as a reference point for all future climate models. LLNL’s Caldwell said the team has “pioneered a performance-portable path for running climate models on exascale machines. Portability has huge advantages because it allows the team to take advantage of compute time on any available machine and provides protection against future disruptive changes in computing architecture.” He added that the impacts of the paper will be felt for years to come, as computing moves into the exascale era. 

“I predict that all climate models are going to need to move to higher resolution in order to provide state-of-the-art predictions,” Caldwell said. “We've been trying to parameterize these sub-grid scale effects for more than 50 years and I'm not sure we will ever get the accuracy we need without instead explicitly resolving the things that matter; computing is giving us the ability now to do that. Our work has received a ton of excitement from a lot of different centers looking to move to these global cloud-resolving models. I think this class of model is just going to become more of the norm and the expectation of people looking for accurate climate predictions.”

By virtue of winning the prize, the team also receives a $10,000 award provided by its namesake, Gordon Bell.