How an inexpensive CDI OOB system could have prevented the Facebook $100,000,000 network outage

Facebook had a very public six-hour outage. We are being told, it wasn’t the network, it was a network engineer. Someone fat fingered a BGP routing configuration information, which in turn took down the entire network, including the physical access to the building where the routers and servers were located.

The network, prevented itself, from being fixed. With all the talk about “virtual”, and “cloud”, everyone forgets that it all comes down to lots of wires connecting lots of appliances with lots of configurations.

This is exactly why you need Secure Out of Band Management to access your network.

Secure Out of Band Access is a secure, secondary, method of access to the key elements in the network for maintenance, in case the primary access to the network appliance is unavailable. OOB appliances connect to the console interface of routers, firewalls, network appliances, etc. They can also power cycle the managed devices remotely.

If a router, firewall, network appliance, etc. gets misconfigured, or goes sideways on its own, it will usually cripple the “in band” access to that area of the network. Without OOB, the only option is to have someone physically onsite to reconfigure, power cycle, or replace the device. Unfortunately for FB, the network also ran the physical access to the building where all the equipment was located.

Secondary access would be cellular LTE, analog modem, or secure internet. All of these methods must be secured to prevent “back door” attacks on a network. CDI has FIPS 140-2 validated encryption OOB products to address this.

CDI’s newer internet based Secure OOB, utilizes certificate based OOB devices, which reach out securely through the firewall to a central certificate based, C2 server. This system creates a full time secure OOB private network using a dynamic internet connection to the remote sites.

These simple architectures prevent $100 Million dollar outages while keeping the network security policies in place.