Network Outage Finger-Pointing | Smart Device Design

What to do then your providers blame each other for a network outage

Network Outage Finger-Pointing

The Chief Information Officer had run into this sort of problem before. His Network Manager was telling him that the leased line provider had an outage on Line AB. The leased line provider was telling him that there appeared to be something wrong with Port 1 on Router A. What he hoped would be a productive information-gathering meeting was turning into an exercise in finger-pointing with both sides indicating that they checked and rechecked their work. So, quite seriously, the two sides were squared off, both convinced the other side had messed something up.

Sometimes it's hard to know where a problem originates.

There was a loss of connectivity between two routers

Because the Chief Information Officer had run into this problem before, he knew exactly what to do. He proposed an experiment. The idea was to connect Leased Line AB to Port 2 and Leased Line AC to Port 1. If the problem remained on Lease Line AB and not on Lease Line AC, then the problem was with Leased Line AB. If the problem moved from Leased Line AB to Leased Line AC, then the problem was with Port 1 of Router A. Both sides eagerly agreed to experiment. This was just the sort of evidence they needed to show that they had done their jobs correctly. Of course, Router A’s configuration would need to be temporarily modified to maintain consistency with the IP addressing scheme in place. Fortunately, this was a simple modification, and the Network Manager had the changes ready within 15 minutes. All that was needed then was to swap the cables and reboot Router A.

To avoid finger-pointing, it is sometimes necessary to gather more information.

An experiment was proposed to isolate the problem

The result was that the problem remained with Leased Line AB. Fortunately, the Leased Line Provider was a reasonable guy, and he was quick to accept what this new evidence meant. He reviewed the provisioning of Lease Line AB for the third time, comparing each parameter with the parameters of Leased Line AC. These two lines were supposed to be provisioned identically. When he found one parameter that was not identically configured, he knew he had found the problem. This was quickly corrected and full connectivity was restored to the wide area network.

The problem was with the provisioning of the leased line.

The experiment indicated the source of the problem

If the Leased Line Provider’s ego was bruised, he didn’t show it. In any case, everyone was relieved that the problem had been solved.