New commodity supercomputing clusters deployed at NNSA labs claim Top500 spots

Three of the newest National Nuclear Security Administration (NNSA) commodity computing clusters recently deployed at Lawrence Livermore National Laboratory (LLNL) and Sandia National Laboratories (Sandia) are among the most powerful supercomputing systems in the world, Top500 organizers announced Monday.

Funded under the second Commodity Technology Systems contract (CTS-2) by NNSA’s Advanced Simulation and Computing (ASC) program, the machines sited at LLNL — named Dane and Bengal — began performing key modeling and simulation functions for the NNSA’s stockpile stewardship mission in mid-2023. Each system is built by Dell Technologies and powered by 4^th Gen Intel Xeon Scalable Processors coupled to Cornelis Networks Omni-Path high-speed networking.

The debut of Dane, a 7.041 petaFLOP (slightly more than 7 quadrillion calculations per second) cluster at No. 108 and the 6.134 petaFLOP Bengal at No. 129, brings the total of LLNL-sited systems on the Top500 List to 11, the most of any supercomputing center in the world. A third new CTS-2 system, sited at Sandia and named Stout, reached 8.987 petaFLOPs on the LINPACK benchmark used to determine the rankings, earning it 87th place on the list. Each system attained 89% or greater efficiency on LINPACK. Top500 organizers unveiled the updated bi-annual list of the world’s most powerful computers at the 2023 International Conference for High Performance Computing, Networking, Storage and Analysis (Supercomputing) in Denver. 

Since their deployment for mission-critical work, Dane, Bengal, Stout and a fourth CTS-2 system named Amber at Sandia, have displayed improved performance and efficiency over the previous generation of commodity systems (CTS-1), with NNSA researchers reporting initial speedups of four- to fivefold across a wide range of high-performance computing applications.  

“The deployment of these first CTS-2 platforms provides a significant addition to the NNSA labs’ simulation environments in support of our national security mission,” said Matt Leininger, CTS project lead for the NNSA tri-labs (LLNL, Sandia and Los Alamos National Laboratory). “We are proud to continue the NNSA ASC tradition of excellence in high-performance computing and U.S. technology partnerships at LLNL, Sandia and throughout the NNSA complex.” 

The commodity-technology-based systems are the “workhorses” of the NNSA and support the day-to-day simulation workload while reducing costs by standardizing hardware and software across the NNSA labs when coupled with the LLNL-led Tri-Laboratory Operating System Stack (TOSS) and Tri-Lab Common Environment. The CTS-2 platforms allow NNSA’s more powerful Advanced Technology System supercomputers, such as the current Sierra system and future El Capitan exascale system at LLNL, to focus on the most complex problems critical to NNSA’s Stockpile Stewardship Program.

Each CTS-2 system is made of building blocks called “scalable units” (SUs) representing about 1.5 petaFLOPs of computing power apiece. The SUs design allows the NNSA laboratories to fine-tune system performance depending on mission need or programmatic budgets. Each SU is built with Dell PowerEdge C6620 and R760 servers that utilize 4^th Gen Intel Xeon Scalable processors.

The 4th Generation Intel Xeon Scalable processors are equipped with purpose-built accelerators that can elevate HPC workload performance and power efficiency by offloading tasks to these acceleration features. Intel Advanced Matrix Extensions (Intel AMX), one of the processors’ built-in accelerator engines, can transform the large matrix math calculations that are at the heart of deep-learning workloads into a single operation, making them critical for delivering performance across workloads where HPC and AI converge.

“These processors are ideal for building and deploying general-purpose AI workloads with the most popular AI frameworks and libraries. These capabilities will make it possible for the engineers, researchers and scientists at Lawrence Livermore National Laboratory and Sandia National Laboratories to simulate complex problems critical to the nation’s national security,” said Deepak Patil, corporate vice president and general manager for Accelerated Computing Systems & Graphics at Intel. “We’re proud to support the labs with solutions that help accelerate the time and effort to analyze future and existing data for their focus areas.”

Each system also is outfitted with 200Gbps Omni-Path Express high-speed networking from Cornelis Networks, providing CTS-2 with a highly scalable fabric with open-source software fully integrated with TOSS. 

“With CTS-2, the team at Cornelis is excited to once again support delivery of the workhorse commodity systems of the NNSA, having previously delivered the high-performance networks for the second generation of Tri-Lab Capacity Clusters (TLCC2) and CTS-1,” said Gunnar Gunnarsson, vice president of solutions delivery and support at Cornelis. “Working closely with our ecosystem partners, Dell Technologies and Intel, Cornelis is pleased to enable leading cluster performance and efficiency at scale for the tri-labs’ mission-critical workloads with our Omni-Path technology. We look forward to continued CTS-2 deployments, including introducing our new 400Gbps CN5000 Omni-Path product family as part of a future architecture refresh.”

Several additional CTS-2 systems will be deployed at LLNL and Sandia in the first half of 2024, in support of various NNSA programs leveraging the CTS architecture and procurement developed by the NNSA ASC program.