That was fun

Well, we had one of the first real tests of the dual redundant set-up in Telehouse today - where our main broadband links, data SIMs, and half our Ethernet customers connect.

We are upgrading core network switches so we have more ports for additional links (adding talk talk lines, and extra peers).

As you can imagine, physically replacing a core switch could be somewhat disruptive!

The design uses an "A" and a "B" side. We configured L2TP links to all be on one side a couple of days ago. We were then able to carefully shutdown BGP sessions to move all traffic off one side at a BGP level. This allowed the switch to be removed, and replaced.

It is always tricky picking when to do stuff like this. Middle of the night is all very well, but that is much harder to have people available to monitor and fix things. In this case we were quite rightly confident that this would work with little or no disruption. The main trick here is taking one step at a time, carefully, and checking everything.

Well done to Andrew, Paul and Alex on this. Even though it pretty much went to plan, this is always stressful work.

The result is that we managed to replace the switch with no impact on our broadband lines, data SIMs or Ethernet customers. In theory there would not have been one dropped packet because of this, and it looks like things did indeed work as per theory. Pings running over the system showed none dropped, as expected.

Of course, just to throw a slight spanner in the works, BT manage to disconnect most of Scotland shortly before we started. We can only guess that they are not quite as careful as us, and we have seen that they have single points of failure in their network.

We plan to do the other switch, probably in a few weeks time. The only change being that we are now monitoring one of our wholesale customers who managed to have their backup link down during the work, which had the result that you might expect, albeit only for a few minutes. We may show them how to set up nagios.

As I say, I love it when a plan comes together - you can rely on the A&A team :-)