In his Nobel Prize acceptance speech in 1980, genomics pioneer Fred Sanger described science as “a voyage of discovery into unknown lands, seeking not for new territory but for new knowledge.” It’s fair to say the renowned chemist’s journey was more eventful than most.

First awarded a Nobel in 1958 for his work on identifying protein structure, Sanger repeated the trick 22 years later, picking up the prize for developing the first DNA sequencing technique, a key breakthrough in advancing the field of genomics. He remains one of few scientists to win the Nobel Prize twice in the same category, and his technique, modestly entitled Sanger Sequencing, is still used in many labs 45 years later.

Another of Sanger’s enduring legacies is the research institute that bears his name. Based at the Wellcome Genome Campus, just a few miles from the Cambridge University labs where Sanger did much of his groundbreaking work, the Wellcome Sanger Institute undertakes genomics and genetics research projects and is one of the largest independent research institutions in Europe. Established in 1992, it was part of the Human Genome Project, the 13-year international effort to map the first human genome that concluded successfully in 2000. Twenty five years later, scientists at the Sanger Institute can perform the same task that took more than a decade to gold standard in less than 12 minutes.

Such a dramatic scientific leap has come with vast amounts of accompanying data, that needs to be stored, processed and analyzed, so the institute has been building out its digital infrastructure at a rapid pace to cope with advances in sequencing.

Not only has the Sanger Institute doubled the amount of GPUs available to its researchers, but it has managed to do so while cutting the energy used in its data center by a third. DCD took a trip to the Genome Campus to find out more.

Gene-ius

The increased volume of work being undertaken at the campus has exceeded the expectations of Simon Binley. As the Sanger’s data center operations manager, Binley oversees the institute’s IT infrastructure, and says the last five years have been a whirlwind.

“We predicted overall growth, but the sheer amount of sequencing that we’ve undertaken since 2019 is staggering,” Binley says. “We have to keep up with that, and it has been challenging. We’ve deployed more software, more storage hardware, and more compute hardware to meet that demand.”

That period encompassed the Covid-19 pandemic, during which the Sanger Institute was the hub for mapping variants of the coronavirus, sequencing tens of thousands of samples from patients across the UK and beyond.

While the work being undertaken at the institution now is somewhat more routine, it does require a lot of compute power and a flexible IT estate. Sanger is constantly investing in new sequencing machines from a variety of vendors, including Illumina, PacBio, and Ultima Genomics, and these all have their own specific software requirements, Binley says.

“There’s been a synergy, as the sequencing technology has advanced at almost an identical rate to the IT technology,” he says. “That’s been good from our perspective, because if sequencing had outpaced IT, we would have been in trouble.

“When it comes to sequencing, we’re technology-agnostic, and many of the groundbreaking machines that are coming through are using newer software, so we have to incorporate these new elements and create a pipeline to support that scientific activity. We’ve invested heavily to keep up with demand.”

With the campus’s sequencing machines producing up to 4TB of data a day, Binley and his team are kept busy. They have a 4.5MW data center on-site, with more than 400 racks across four data halls, utilising both air and liquid cooling systems. With artificial intelligence playing an increasingly important role in life science, the campus has joined the rush to secure the latest hardware, and has increased the computational power available to its researchers since the pandemic by 50 percent.

Its shiny new toys include an Nvidia DGX system, which the institute is using alongside the vendor’s Parabricks software for genomics research. An Nvidia blog last year detailed how Sanger scientists deployed the DGX system to carry out a specific type of analysis using the institute’s proprietary CaVEMen workflow, which is designed to detect mutations in cancer genomes. By using DGX, the researchers were apparently able to reduce runtime 1.6x, costs 24x, and energy consumption by up to 42x, compared to running the process on the standard set-up of 128 dual-socket CPU servers.

Efficiency gains

Such efficiency gains are vital in a research environment, where funds are often limited. As well as contributing to the data center’s overall sustainability, keeping costs low allows more money to be funnelled to the science that is at the core of the Sanger Institute’s mission.

Binley and his team have been working with EfficiencyIT, a specialist in data center design, to ensure the institute’s data centers work as effectively as possible. By installing Schneider Electric’s EcoStruxure IT Advisor, on-premises data center infrastructure management (DCIM) software, and more than 300 custom-designed APC pack power distribution unit (PDU) systems, the companies claim the data center’s energy usage has been reduced by 33 percent.

A host of sensors were installed across the data center to gather environmental and mechanical information, which was then analyzed using machine learning-based data analytics systems to help remove stranded capacity and manage operating expenses.

Nick Ewing, managing director of EfficiencyIT, says: “Simon needed more information about the data center, so we designed a custom PDU with Schneider that we could fit without any downtime. We then have physical sensors on the cooling units and racks so we understand the core metrics around the cooling environment, and we have full electrical metering and monitoring on all the electrical subsystems and PDUs.”

On top of this physical infrastructure, a series of “virtual sensors” has been deployed using software, meaning the Institute’s IT team “can create virtual power groups for different systems,” Ewing says.

These virtual power groups have been central to the efficiency effort, he adds. Referring back to the variety of sequencing machines deployed at the campus, Ewing explains: “A lot of the researchers at the Sanger Institute are making decisions about which hardware platforms to use, so if we have two platforms that are doing the same types of sequencing, we can monitor those and see which platform performs most efficiently.

“That’s where the institution is probably seeing most of its energy savings – choosing the technology platforms that support the science as efficiently as possible.”

Mapping the future

The virtual sensors, managed via the EcoStruxure DCIM, are also playing an important role as the data center grows, Ewing says.

“They help with capacity planning as more infrastructure is moved into the environments,” he explains. “It helps figure out where there is spare network or power capacity, and whether infrastructure will fit within the physical boundaries of the data center.”

The sensitivity of some sequencing equipment means it has to be located in close proximity to the servers processing its data, Ewing adds. “We’re talking 50-150 meters away, so space planning and capacity are vitally important,” he says.

This requirement for compute to be in the vicinity of the lab is one reason why the Sanger Institute has no plans to embrace the cloud, Binley explains.

“Proximity is critical,” he says. “We’re offloading all that information into the data center for storage and rapid primary analysis. The cloud has had a significant impact on the world of IT, but it isn’t for us, not least because we would need 124 petabytes of storage – can you imagine how much that would cost? We’d constantly be putting data in and taking it out, so the ingress and egress charges would be significant.”

Instead of running up the world’s largest AWS bill, the Sanger Institute intends to expand its on-premises data center infrastructure as and when it proves necessary.

“The campus has 11MW of power to it, and we have 4.5MW available for the data center, which we’re not fully using yet,” Binley says. “That’s a massive asset as we look to expand and grow, and we expect to do that again over the next few years.”

Binley’s cause could be helped by plans to expand the Genome Campus. Permission has been granted for the first phase, which will see two new buildings constructed. It is hoped the site will eventually grow to support 9,000 jobs, rather than the 3,000 currently connected to its activities. More than 1,500 new homes are also being built in the area, as well as a new electricity substation.

“They’re talking about bringing in another 33MVA across the road, which is a significant amount of additional power,” Binley says. “As AI takes hold, we anticipate a greater need for power. I would say it could double, or even treble over the next five to ten years, though I’ve learned with science it’s difficult to predict what will happen.”

That brings us nicely back to Fred Sanger’s speech at the Nobel Banquet in 1980, where he added that science should “appeal to those with a good sense of adventure.” Certainly, it seems that, when it comes to AI, the Sanger Institute’s adventures are only just beginning.

The feature first appeared in the DCD Life Sciences supplement – register here to read the entire supplement, free of charge.