Blog

Built in the USA

How a Major U.S. Utility Cut Network Recovery from Hours to Minutes

articles
Avatar of Adam Cady

Adam Cady

cdi product

From Hours to Minutes: How a Major U.S. Utility Accelerated Network Recovery with CDI Secure Out-of-Band Management

Customer: One of the largest electric utilities in the U.S., with 900+ substations across a service territory of roughly 35,000 square miles.

Challenge: No access to network equipment when the production network went down; technician response often took 10+ hours, outages typically lasted 5+ hours, and the team logged hundreds of truck rolls a year.

Solution: CDI FIPS 140-3 validated, LTE-based Secure Out-of-Band Management with remote power control at approximately 1,000 sites.

Results: Mean time to recovery cut from 5+ hours to minutes; truck rolls reduced from hundreds a year to fewer than 50; full audit trail of administrative access.

The challenge: flying blind when it mattered most

The utility's network operations team managed routers, switches, firewalls, and serial-connected devices across more than 900 substations, spread over a service territory of roughly 35,000 square miles. Many sites sit hours from the nearest technician. Day to day, the in-band management network worked well. The problem was the days it didn't.

When a firmware update failed, a configuration change dropped a routing adjacency, or a circuit went down, the team lost visibility into the affected site entirely. The tools they used to diagnose the problem ran over the same network that had just failed. Engineers could see that a site was dark, but not why.

Recovery followed a familiar, expensive pattern:

  • Dispatch a field technician. Between scheduling, dispatch, and travel, getting someone on site often took more than 10 hours.

  • Talk the technician through console commands by phone, because the network engineers who knew the configuration were back at the operations center.

  • Power-cycle equipment by hand when a device hung.

A typical outage lasted more than 5 hours, and the team logged hundreds of truck rolls a year, many for problems that could have been fixed remotely. Leadership also had a growing compliance concern. Any new remote access path into substations had to meet strict security expectations for critical infrastructure. Consumer-grade cellular gateways and internet-exposed remote access tools were not an option.

The solution: a secure, independent path to every site

The utility deployed Communication Devices, Inc. (CDI) Secure Out-of-Band Management at approximately 1,000 sites, covering its substations and other remote locations. Each site received a CDI unit connected directly to the serial console ports of its critical network equipment, along with a CDI power control module for remote power cycling.

Four design choices made the deployment work for a security-conscious utility:

  • Independent connectivity. Each unit connects over LTE on a private APN, so management traffic never touches the public internet and stays available when the production network is down.

  • Validated security. CDI's equipment is FIPS 140-3 validated at the device level (CMVP certificate #5465), which helped the project clear the utility's security review.

  • Built-in two-factor authentication. Every engineer signs in with a named account and 2FA, giving security and compliance teams a clear record of who accessed which site and when.

  • Remote power control. Serially controlled power modules let engineers hard-reset a hung device from the operations center instead of sending someone to the site.

Because CDI units provide direct serial console access, engineers see exactly what the device sees, including boot messages and error output, even when its network interfaces are down.

The results: faster recovery, fewer truck rolls, better visibility

With CDI in place across the network, the operations team changed how it handles outages. Recovery that once took more than 5 hours now takes minutes, and truck rolls for network faults dropped from hundreds a year to fewer than 50.

Measure

Before CDI

After CDI

Mean time to recovery

5+ hours

Minutes

Time to get help on site

Often 10+ hours

Remote access in minutes

Truck rolls per year for network faults

Hundreds

Fewer than 50

Sites reachable during a production outage

None

All ~1,000

Record of administrative access

Partial

Every session, by named user

Visibility when it counts. When a site goes dark, engineers now connect to the console within minutes and see the actual cause, whether a failed upgrade, a bad configuration, or a hardware fault. Diagnosis no longer waits for someone to arrive on site.

Recovery without travel. Most incidents are now resolved entirely from the operations center: roll back a configuration over the console, or power-cycle a hung device through the power module. Field technicians are dispatched only for genuine hardware replacements.

Safer changes. Knowing they have a guaranteed path back in, engineers schedule firmware upgrades and configuration changes with more confidence. If a change goes wrong, recovery is minutes away.

Audit-ready access. Named accounts, 2FA, and session logs give compliance teams a clear answer when auditors ask who accessed critical infrastructure equipment, and how.

In the customer's words

“Using our new CDI cellular devices, we are now able to quickly and remotely configure a CDI unit on site. We are now also able to utilize these devices to help us turn-up new sites since no infrastructure is required. This has helped expedite new site deployment as well as facilitate troubleshooting”

Network Engineer
Top 10 largest US Utility Company

Bring the same visibility to your network

CDI builds U.S.-made, FIPS 140-3 validated, LTE-based Secure Out-of-Band Management and power control solutions for organizations that cannot afford to lose access to critical infrastructure. Whether you run substations, data centers, or remote field sites, CDI gives your team a secure path back in when the production network is down.

To see how CDI can shorten recovery times in your environment, call 973.334.1980.

Related Tags

Share this article

Related Content

cdi product

When Automation Fails: The Human Control Plane

Avatar of Adam Cady

Adam Cady

Why autonomous networks need secure, independent access to keep recovery in human hands.

  • United States Office

  • 85 Fulton Street Boonton, NJ 07005
  • +1 973-334-1980
  • +1 973-334-0545
  • info@commdevices.com

Connect with us

© 2023 Communication Devices, Inc.