FB9000

I know techies follow this, so I thought it was worth posting and explaining...

The FB9000 is the latest FireBrick. It is the "ISP" high end model we do. We do smaller models like the FB2900 as well, but FB6000 and now FB9000 are aimed at ISPs and the like. It is what A&A use.

You can see a lot more here: https://www.firebrick.co.uk/fb9000/

But why now - the FB9000 has actually been around a while?

We are not like other companies!

When we launched the FB9000, we obviously started using them ourselves, in A&A.

We hit some snags, some random crashes, we backed off, we found a release of the code that worked and was stable, but that does not address the underlying cause. Why did some releases crash? So we were able to continue with a good set of working LNSs on a somewhat aging reliable release of code. But it meant some inconvenience for our customers along the way when we tried other code. We do not like that! So we massively backed off.

Thankfully some devices, notably BGP routers with VRRP, which annoyingly crashed far less often, can recover in literally 1/10 of a second. So they were good test cases for new code without upsetting customers. An LNS does not recover as well as all users need to reconnect and that can take minutes, depending on their router.

You would not believe the details behind the problems, seriously, it is crazy, and I am not even going to try to explain it here. There may be a really detailed technical blog post by the FireBrick team in time. Suffice to say this snag held us back something like a year.

Now, we could have plowed ahead, and sold loads, but we were really careful not to. A couple of ISPs trust us enough to solve it that they have the stable code release running and did buy some. Thank you. They did so very aware of the issues and have been fine on the stable code release.

It takes time

The issue is that the fix literally takes months to be sure it is a fix. And at A&A we have been doing very very careful staged upgrades to LNSs to prove this, with a lot of staff working during the night to manage this (well mostly one, thanks Andrew). This has taken months even after we think we have nailed the underlying issue. Thank you to all of the staff involved.

We are now at the stage we can probably say it really is fixed, at last. But it is one of those things which are a problem - you cannot be 100% sure until it doesn't crash. Yeah, when exactly is that?

Chasing ghosts

We really are pretty damn confident now. The issue is that, as an engineer, you want to find the smoking gun. This issue is a horrid mix of hardware quirks that even the chip manufacturers cannot explain, and some very very subtle hardware initialisation that has impacts days, weeks, even months later in running code. We have found some concrete issues, well, things not quite 100% as they should be, but not the causal link you want between such things and the problems we saw. And this is not for a lack of trying - every time we thought we found the cause the team have tried hard to break it in a repeatable way. To overdo what we may possibly have done wrong.

A product we can sell

This has always been an awesome product, and any other manufacturer would have fired off the marketing team years ago for sell - sell - sell.

We finally have something we can say with a lot of confidence works well. Does the job, and does it well.

There is more

The FB9000 is awesome, and if you are an ISP you really want one - they have some unique features that really gives A&A an edge which you too could enjoy.

But we are working on a next generation for the smaller units, the FB3100 to succeed the FB2900. It too will take time, and we hope none of the same issues. The FB2900 is also awesome, and there are some offers I think on the pricing soon.