HPCAT goes remote

Before the COVID-19 pandemic, HPCAT, a Chicago-based research consortium to advance high-pressure science in multidisciplinary fields using synchrotron radiation, hosted as many as 750 experimentalists each year—including numerous Lawrence Livermore National Laboratory (LLNL) teams.

The consortium operates Sector 16 at the Advanced Photon Source (APS) at Argonne National Laboratory (ANL), the largest synchrotron facility in the United States. Sector 16 comprises four active beamlines and an array of X-ray probes optimized for high-pressure research. APS-HPCAT is fully supported by the National Nuclear Security Administration (NNSA) and governed by a joint NNSA–Department of Energy Office of Science executive board, with APS providing operational oversight and HPCAT director Nenad Velisavljevic—an LLNL employee—overseeing the implementation of scientific and strategic direction of the facility.

On March 17, 2020, all work at HPCAT came to a standstill. “During the initial shelter-in-place, we were unable to work on site and we didn’t have an option to host remote experiments,” Velisavljevic said. “HPCAT provides important mission support to NNSA labs and many academic partners, including numerous students and postdocs. We said ‘Let’s not wait. Let’s look at all options and consider taking HPCAT remote.’”

Prior to COVID-19, HPCAT had never hosted a remote experiment. In fact, it seemed practically impossible, as HPCAT high-pressure experiments require significant in-person interaction. Between March and June, Velisavljevic’s team collaborated with the HPCAT research community to figure out benchtop logistics, including implementing reliable computer-controlled instrumentation and diagnostics, automated procedures and digital access permissions to enable remote computer login and experimentation.

To take HPCAT remote first required the evaluation and implementation of a robust, commercially available server possessing enhanced latency characteristics, making it flexible and reliable enough to allow researchers to perform experiments using remote-controlled cameras to “see” their work.

To make the remote setup more efficient, the team also developed an automated magazine containing two experiment samples so that the skeleton crew of HPCAT on-site staff could load test samples less frequently and reduce the number of times the experimental door had to be opened and closed. “Opening the experimental door is a five-minute process,” said William Evans, physics division leader in the Physical and Life Sciences (PLS) Directorate and a long-time HPCAT experimentalist. “One of the welcome improvements of remote automation is that we don’t have to open and close the door as often. It doesn’t seem like much, but those minutes add up. The HPCAT team went a long way to fully realize many efficiencies.”

In mid-June when limited operations were permitted by the State of Illinois, 50 percent of HPCAT staff members were allowed on site with mandatory masking and social distancing protocols. With additional funding from the CARES Act, APS/ANL-DOE and NNSA core funding, it took about a month to install remote-controlled cameras, and about three weeks to implement and test the systems. Then, following a scheduled maintenance hiatus, HPCAT began its first-ever remote user experiment on July 12 and was fully operational by Oct. 1.

Today, HPCAT has about 15 staff members, including four on-site assistants and four stations running experiments simultaneously by as many as seven remote experimenters. Some experiments still require in-person monitoring, but at least half of the experiments can be run remotely.

“We don’t do the experiments for others,” Velisavljevic explained. “We receive a sample package and set up the experiment, then the experimenters log in and are able to fully control and conduct the experiment, monitor its progress via cameras and receive data in real-time.”

Velisavljevic is especially proud of the way the HPCAT staff stepped up to the challenge of taking HPCAT remote and grateful to the initial users who were willing to contend with the challenges of the new system. “Our first users were venturing into uncharted territory,” he noted. “They had to be patient with log-ins, getting blocked by the firewall and going to the IT folks to get access to the server.”

For Kanani Lee, a PLS research physicist who has conducted three experiments using HPCAT’s remote access, “telecommuting” to HPCAT has been challenging and rewarding. “At first there were a few technical bumps like network connectivity and data transfer speed,” Lee said, “but these glitches have mostly been ironed out. Beamtimes are typically 24/7 endeavors where you spend a couple days exhausted and not eating very well. I’m appreciative of the remote beamtime because I can sleep in my own bed, and the long hours don’t feel so long. My family appreciates it too. I was skeptical that it would work as well as it has. That goes to show you how fantastic the beamline support staff is.”

Today, HPCAT is working at full capacity and providing approximately 60 percent of available experimental capabilities, but steadily pushing to increase and broaden access. “The remote operations infrastructure will be useful even after COVID restrictions lift,” Velisavljevic said. “Having a remote option will allow the user community to still perform some experiments remotely and reduce the travel burden. In some cases, the remote setup also presents significant, additional value as a potential pre-experiment test run that can be used to fine-tune plans and procedures before coming on-site to execute more complex experiments”.

“Being able to perform these experiments remotely saves money and is also a work-force multiplier because it allows us to shift parameters, give teams more flexibility and improve efficiencies. These experiments take 24 hours to complete; people get tired. But now, they can have a larger team supporting an experiment,” Evans said. “But nothing beats being in person with other scientists,” observed Velisavljevic. “The best scientific ideas often come when we’re chatting in the hallway or over a cup of coffee. We really don’t want to eliminate that.”