Growing pains

When we first started we had a few months when we actually stopped taking on new customers - it was all down to a silly long lead time and some other issues with a new BT pipe.

Back in those days there were 2Mb/s, 8Mb/s, and 34Mb/s pipes, and we were going to a 34Mb/s pipe at the time. We then moved to 155Mb/s and then two of them. That took a whole rack of BT kit!

Finally we moved to Gb/s host links, and we currently have many Gb/s fibres to both BT and TalkTalk. Surprisingly these are getting full quicker than we expected.

Well, not quite. We expected that by start of 2016 we would need more capacity in our core network to link all of this together and we started planning early last year. We have even taken on a highly experienced network architect to help us manage this all.

We have been taking things slowly, one step at a time, and cautious not to break things, but then things did not quite go to plan. A fibre install has taken a lot longer than expected, and then turned out to have some stupid jointing issues which have taken some time to fix. We have had to invest in some rather expensive but very fast switches with lots of 10Gb/s ports, to make our new network in two London data centres with 80Gb/s cross connect.

The problem is that we are not quite there yet - we are about a week away before we can start to actually use the new network and links, and right now we have higher than expected traffic causing some congestion in the evening for some of our TalkTalk customers and even some of the Ethernet customers.

This is embarrassing. I know lots of ISPs have congestion, and for some it is simply part of the business model, but we pride ourselves in not slowing down in the evenings and not being a bottleneck, and we have failed in that for about a week now in the evenings on some lines.

So it is a trade off to get stuff moving quickly, and to make 100% sure we are doing it right. I think we have the right focus now, and assuming some optics arrive this week we should be able to have the new cross connect links between the data centres under test this weekend. There is some planned work which will cause some disruption early one morning at the weekend, but once that is done we can start fixing these congestion issues once and for all.

In the mean time I have delayed the launch of the Home::1 ADSL Terabyte service - a package which I hope it will mean lots more customers. Right now I would rather our existing customers have the best possible service than just piling more people on. We also have done some juggling of links to try and alleviate the problem for tonight, and expect to be able to move more traffic to another router in the next couple of days as well. If all goes well, we will be OK this week and can get started on the long term solutions next week.

This latest expansion should last a few years, but even so we expect that we will have to add more LNSs and more routers as time goes by.

Fortunately we don't have Theresa May poking her nose in as it would cause a lot of problems trying to manage this network expansion and manage snooping for the spooks at the same time. We can only hope that we never have that problem!

Thank you all for your patience.

P.S. New switches and links going in on Sunday morning (13th)