PNNL

Exascale Grid Optimization (ExaGO), a power grid simulation and optimization platform developed by Pacific Northwest National Laboratory (PNNL), is the first of its kind to run on Oak Ridge National Laboratory’s (ORNL) Frontier, the first supercomputer in the world to reach exascale.

Frontier, which was launched this Spring, can calculate more than 1 quintillion operations per second and is 10 times more powerful than its predecessor, Summit. ExaGO uses HiOp, an optimization solver created by Lawrence Livermore National Laboratory, PETSc (Portable, Extensible Toolkit for Scientific Computation), a high-performance library developed at Argonne National Laboratory, with supporting software from ORNL, and wind data provided by National Renewable Energy Laboratory.

The test runs, which were completed between April and May, confirm the ability of ExaGO to accurately model the behavior of the grid under highly complex scenarios – such as severe storms or cyberattacks – at unprecedented speed.

The whole test run took 20 minutes from start to finish.

This is important because operators rely on 30-minute windows to assess the performance of the grid and deploy resources in time to prevent costly outages and blackouts, such as those experienced across the East coast caused by Winter Storm Elliot in December. ExaGO can also be used to help manage load volatility caused by integrating distributed energy resources onto the grid like wind and solar.

“We’re seeing a lot of renewable energy coming and growing incidences of natural disasters, such as wildfires and hurricanes, which all impact the ability of the grid to reliably deliver critical service to customers,” said Shrirang Abhyankar, senior optimization scientist at PNNL, “ExaGO will help operators plan for emergencies and respond much faster.”

ExaGO was developed under the ExaSGD project, which involves five national laboratories and Stanford University and is funded by the Department of Energy’s (DOE’s) Office of Science Exascale Computing Project. The first stable version of the software was launched last year.

Why exascale?

The nation’s power grid – from generation to transmission – is a large and highly complex system with countless interdependencies and possible points of failure. It’s also experiencing rapid technological changes, which exposes the grid to greater risk.

The stakes are high. A prolonged outage could put the lives of people at risk, disabling life-sustaining medical equipment or leaving them exposed to extreme temperatures. Protecting the system from failure is a high priority for the DOE. Unfortunately, current computational tools are incapable of processing data and rendering graphics at the magnitude required to model one of the most complicated systems ever built.

“Frontier will allow us to solve bigger problems that we couldn’t solve before,” said Chris Oehmen, principal investigator on the ExaSGD project and computational biologist at PNNL. “Current computing capabilities only allow for modeling a small part of the entire grid, or evaluating a small number of scenarios, forcing grid planners to make calculated guesses and close approximations.”

But when people’s lives and livelihoods are on the line, that’s not good enough. We need many high-fidelity simulations in near real-time that can adequately accommodate the quick changing pace and variability of a grid with renewable energy and other new edge devices.

ExaGO removes blind spots by giving grid planners the full picture of what could happen under any number of scenarios.

“If you have a hurricane that’s coming, you need to know where to send trucks and transformers immediately. You don’t have a week,” said Oehmen. “A lot of what we’ve been doing is enabling ExaGO to take advantage of these faster computers.”

How it works

ExaGO was designed to work on machines like Frontier from the outset and can be deployed at large scale on computers using hardware accelerators, such as graphics processing units (GPUs). Compared to traditional processing units, GPUs can perform computations at much faster rate and with less power than traditional processors but require more elaborate mathematical algorithms to harness that computational power. These features are necessary  to model stochastic grid behavior, which can be analyzed statistically but cannot be predicted precisely.

“We are pushing the envelope of the number of scenarios that we can study,” said Abhyankar.

“The current standard is that you only model and study n-1 scenario, which often means those elements in the grid that are directly impacted or go offline, but that’s limiting. What exascale allows us to do is push beyond the n-1 and study all the combinations of scenarios under a variety of possible weather conditions that are statistically available at a much faster speed,” commented Abhyankar.

Preparing for a successful launch

This April, the team lined up ExaGO in a queue to wait their turn to run on Frontier and started by running smaller jobs. They progressively built to an exascale simulation of a synthetic power grid the size of the Western Interconnection, which is the section of the electric grid servicing everything west of Colorado and reaching the Canadian provinces of British Columbia and Alberta.

Starting small allowed them to work out any flaws in the system before launching the large-scale simulation they’ve been preparing for years. During the simulation they introduced several contingencies that mimicked grid behavior under a hurricane and other events, like bringing down transmission lines, increasing load volatility caused by excessive wind generation, and shutting down power plants.

“We ran tens of thousands of these what-if scenarios at the scale of Western Interconnection and observed the impact on the grid, such as localized outages or grid-wide blackouts caused by cascading failures,” said Abhyankar.

Now the ExaGO team will validate the output of ExaGO to ensure it reflects actual operating conditions.

ExaGO can also run on smaller systems—including laptops—and is available for general use through open-source licensing on GitLab. For more information about ExaGO, contact Shrirang Abhyankar at PNNL.