This reminds me of an Internet outage in the ’90s.

When you buy a circuit from the circuit provider you tell them the type of circuit you want. If it is a big enough circuit they will provide you with alternate routes.

In the days when OC-12 was a “big circuit” one was ordered for post. They had to lay cable up from the south and down from the north. If either segment was “cut” then all the traffic could be carried via the other segment. Redundancy.

Now the magic of buying circuits is that you specify what you are buying “T1” for example. This slow by today’s standards at 1.44Mbit/s. It is so slow that your provider will use different technology to deliver it. In one case it was by two pairs of copper using standard digital technologies. It took two pair bonded together to get the speed wanted. No issue as the equipment for doing the bonding was cheap and easy to use.

The Federal Government decided they wanted an east-west circuit. It was supposed to be a fairly fast circuit. Part of the requirements in the contract was that the circuit be fully redundant. This meant that one path was through the north of the US and the other path was through the south of the US.

All of this documented in the contracts.

At the time the contract was issued this meant that each path was its own physical piece of fiber.

But shortly thereafter technology moved on and it became possible to move significantly more data over the same physical links.

The provider looked to take advantage of this and let a contract for two new circuits, again redundant, again north south paths.

Everybody is happy. Government, provider, and sub provider are all doing the right thing.

The sub-provider is constantly working to improve the infrastructure and upgrade circuits when they can. In the course of this they upgraded a couple of circuits and then re-balanced everything.

The sub-provider was still providing two circuits, the two circuits were redundant. The sub-provider was providing great service and they continued to grow.

Then one day the Internet Segmented into two parts barely connected. Traffic that use to flow over these east-west circuits now had to travel east from the east coast around the world to get to the west coast.

Multiple circuits were down. The primary and redundant backups for the Federal Government were both down. Other internet circuit providers had lost both of their east-west circuits. Whiskey-Tango-Foxtrot!

It seems that over the course of time our sub-provider had been offering very good prices for circuits. They had the latest and truly greatest technology. And as time passed more and more circuits were being carried by this one sub-provider. Their southern cables were the fastest in the country. Their northern cables not so much so they had moved some circuits from the northern cable to the southern cable.

All of which was allowed by their contract. The government contract was with thier provider, not the sub-provider.

It seems that a couple of good ol’ boys were out in the swamps hunting and decided to blow of some steam. They decided to use the targets hanging from poles as their point of aim. In the course of their shooting they managed to cut, in multiple places, in the middle of a swamp, that sub-providers main east-west PHYSICAL cable.

Since multiple “redundant” circuits were all running over the same physical cable when that cable was cut all the circuits failed.

It took many hours for that cable to be repaired.

Now to make things even a little bit scarier, my understanding is that the original contracts were let to two different providers with two different cable system. Those two providers at some point started using the same sub-provider and so even though it was two different providers, the redundant system failed.

Our infrastructure is fragile. Timothy McVeigh used an ANFO explosive to take down a building. Think of what would have happened if he had just parked that van on the middle of the George Washington Bridge and gotten into a different vehicle to drive away. Depending on the tamping around that charge it might very well have dropped the bridge right in the Hudson. Even if it didn’t drop the bridge, it would have done enough damage that it would be months if not years before that bridge was fully inspected and cleared for traffic again.

And there is no way that they would be able to inspect every truck that travels over every bridge.