New visualization cluster installed on world’s fastest supercomputer

Nov. 7, 2005

New visualization cluster installed on world’s fastest supercomputer

LIVERMORE, Calif. — The National Nuclear Security Administration’s (NNSA) program to ensure the safety, security and reliability of the nation’s nuclear deterrent in the absence of testing now has one of the world’s most powerful visualization computers installed at Lawrence Livermore National Laboratory (LLNL).

Named “gauss,” the new visualization cluster was built by GraphStream Incorporated, in partnership with NNSA’s Advanced Simulation and Computing (ASC) Program, to support the visualization and data analysis needs of LLNL’s BlueGene/L (BG/L), which is the fastest computer in the world. BG/L’s recent breakthrough calculations relate to materials aging and turbulence, providing new scientific insights needed for NNSA stockpile stewardship.

Steve Louis, LLNL Subprogram Lead for ASC Computational Systems and Software Environment, said, “We realized that our current visualization solutions would be inadequate to handle the massive data that would be generated by BG/L codes. At early meetings, researchers commented that BG/L would produce output so massive it could only be handled by BG/L itself. Gauss provides a solution.”

Delivered in September, the gauss cluster is a 256-node dual-CPU multi-teraFLOP/s cluster supercomputer based on Tyan™ S2895 motherboards with AMD Opteron™ CPUs running at 2.4GHz. In addition to the two CPUs and 12 GB of memory, each node contains a high-end NVIDIA™ Quadro 4500 graphics card, a feature that differentiates gauss from traditional computing clusters. According to Dale Southard, LLNL’s hardware architect for gauss, “If you look at an individual node of gauss, its design is very similar to an ultra-high-end consumer gaming machine – fast CPUs, lots of RAM, and a video card that gamers would die for.”

The Quadro 4500 graphics cards are leading-edge products, similar to consumer model 7800GTX cards, but have twice the memory and provide better performance for advanced OpenGL™ operations like pixel readback. The 3D engine in each Quadro 4500 card can perform more than 150 billion floating point calculations per second. By harnessing these cards in parallel, gauss can visualize extremely large datasets containing billions of triangles in real time. Because gauss is based on Linux, it can run modern software tools such as LLNL’s VisIt, Kitware’s ParaView™, and CEI’s EnSight™, distributing the visualization work across multiple video cards. For other software tools, open-source Chromium cluster rendering middleware provides a way to distribute the rendering load of unmodified OpenGL applications.

The gauss nodes are connected by a Voltaire™ InfiniBand network. This interconnect provides transfer rates of more than 800 megabytes per second to individual nodes. Notably, gauss uses OpenIB™ software drivers to run the InfiniBand network – one of the first clusters of this size to do so, and the first large InfiniBand cluster of any kind at LLNL. According to Southard, “LLNL was able to run MPI jobs that performed more than 1.8 trillion operations per second on gauss within two weeks of receiving the hardware. This rapid deployment and stability is a tribute to the maturity of InfiniBand hardware and Linux OpenIB drivers.”

Along with new technologies, gauss also builds on existing software expertise. Like other clusters at LLNL, gauss runs LLNL's CHAOS (Clustered High Availability Operating System) , based on Red Hat Enterprise Linux. CHAOS provides seamless administration across the entire range of LLNL Linux clusters, and requires no modification of node hardware or firmware. In addition to InfiniBand for intra-cluster communication, gauss shares a 900 TB Lustre™ filesystem with BG/L over 256 separate gigabit ethernet links that can move data in parallel, providing high-bandwidth, zero-copy access to simulation data. The ability to leverage existing software and tools on the newest commodity hardware provides quick, cost-effective solutions to critical NNSA computational problems such as visualization of massive BG/L data.

Founded in 1952, Lawrence Livermore National Laboratory has a mission to ensure national security and apply science and technology to the important issues of our time. Lawrence Livermore National Laboratory is managed by the University of California for the U.S. Department of Energy's National Nuclear Security Administration.