When launching a new product, especially when we are talking a real, electronic, product, there is a lot to do.
There are some time consuming and hard parts in designing schematic, PCBs, metalwork/case, software. There are loads of fiddly bits in terms of cosmetic decisions on colour and layout of the artwork on the front/back panels, and all sorts of things like that.
Then, when you have all of that, there are a few rather annoying things, one of which is the new safety standards we have to follow: BS EN 62368-1:2014.
This is new. Last time we did a product it was way simpler. We still have to do all of the EMC testing, which is pretty straightforward and was done some time ago, but the new safety testing is much more work.
This is, of course, all good stuff. We are good on safety. We have a good solid metal box which won't catch fire! We have a proper physically isolated double insulated power supply. In fact we redesigned the whole power supply from the last model and it allows some addition DC options we will have shortly (at this rate, at the same time as mains versions).
But the last step of safety testing needs a final production sample. Sensible. Sadly that means little details that are usually minor things at the end of the production like the printed fascias, and the quick start guide, are actually necessary for the safety testing! The manual includes mandatory details of power, and humidity, and temperature, and altitude, and so on. Having the manual as supplied ready, and all of the bits to make a final production sample, is needed, and then we have to wait weeks for the final sign off, probably. These people are in no rush, it seems.
So I really thought we would be shipping by now - we have hundreds on the shelf ready to go and customers eagerly awaiting the new product, and we have a fly in the ointment. The damn printers for the fascias. Promises of quick turnaround digital prints ahead of the final screen prints so we have our production sample were clearly outright lies (48 hour turnaround says the salesman). Over a week later we get one side and not the other.
I was really pissed off at that (somewhat shouting at one of my staff even, not his fault). I was literally sat here waiting for news on this - have they finally arrived after promise after promise, and then find they sent only one side and not the other. This one minor supplier has added a week or more to when we can start selling these. I am exasperated.
But, calm down, step back, and let my staff do their stuff. We wait and see when the other side arrives and we can finally send the production sample for final safety testing. It will happen in the next few days. There is no guarantee we pass, but we know we are doing well, and we know the issues and we are good. We are pretty sure of that. I could find there is some other delay in a week to two's time, but we know there can be nothing major.
Once we finally get the sign off, we can do the formal paperwork, declaration of conformance and attach the dreaded "CE" mark to the product and start shipping.
In the mean time, next week or two, we may have a few prototypes for evaluation for some select customers. If you get one, you can consider yourself special.
But what makes the FB2900 special?
FireBricks are a tad special. There are many routers and firewalls and all sorts around now - when we started there were not. But there are actually only a handful of underlying operating systems out there - linux, vxworks, maybe a couple of others. We have our own operating system from scratch and it is all UK designed and built. We control every line of code from scratch and ensure no back doors.
The FB2900 is the latest in the long line of FireBrick products. It builds on the FB2700 but is a lot faster (up to around 750M routed traffic rather than 350M) and has an extra port (a 5th port is SFP).
This makes it truly ready for the latest Internet links, 330M G.fast and FTTP, and multiple bonded FTTC.
The SFP allows true fibre lines, and even some of the new DSL SFP modules to be used directly.
The DC options are interesting - I mean, who wants to use a FireBrick in a car or a truck? Well, you would be surprised, but what of alarm panels? Many alarm panels have 12V DC lead acid backup and these days they need Internet. A 12V FB2900 with SFP DSL could be that backup for a long time. The power requirements are really low. This is actually what I plan personally at home, albeit with fibre SFP.
It also has a few gems that are waiting in the wings. We have a true random number generator that will come in to its own as we launch newer software in the coming months. It even has a hardware AES crypto processor which we really hope to link in to the IPsec code later in the year. The key here is the software updates are free and automatic, so these advances will be in there as soon as we launch them. Obviously we need to do a lot of work on these, so no guarantees yet, but the hardware is built to support them.
Oh, and unlike previous models we actually have a rack/wall mount kit. It does one or two FB2900s in 19" 1U or a wall mount, either way round/up.
*stands in queue*
ReplyDeleteThat is the real sign of an Apple-esk product :-) You are pretty high on the list Neil, believe me...
DeleteI am camping outside right now (honest)
ReplyDeleteLovely to see and hear all about it
ReplyDeleteDo existing Alpha testers get a discount?
ReplyDelete"We have our own operating system from scratch"
ReplyDeleteDo you seriously think that reinventing this particular wheel is a recommendation rather than a strong disrecommendation these days? Operating systems are a) commodities and b) nontrivial ones are far too complex for small groups to hope to maintain in any meaningful fashion. Worse yet, an operating system for modern (highly concurrent) hardware used on only a small number of devices almost certainly contains large numbers of unknown rare bugs and race conditions which have simply never been encountered because the code hasn't been running for long enough. If you were to use a widely-used operating system rather than rolling your own wheel you'd have the benefit of literally billions to trillions of runtime hours on hundreds of millions of systems to help debug the OS for you.
(And the "UK designed and built" is better passed over: having a specific national origin is *not* something which is going to make software better! It doesn't even keep transport costs down :) )
The right defence against backdoors is not to have the software written by a small group which promises it doesn't have any back doors: in fact, if the group is in a single country the jurisdiction in question could easily force the developers to introduce a back door and then lie about it (and, of course, there is no way for any group to prove that this didn't happen). The right defence against backdoors is to have the code as open as possible, so that no trust is needed and interested parties can look at it and *verify* the absence of backdoors (and then flash the code they verified onto their devices themselves, to be sure: the really paranoid will use their own SPI programmer etc). I know not many people actually do this but the mere possibility of things wriggling out and causing PR disasters when people do this and find backdoors serves as a disincentive for those who would like to quietly introduce backdoors. Having the developers in many countries (so that at least one is unlikely to have legal retaliation as an impediment to her analysis of the code) is also beneficial.
I don't understand why you don't use a normal OS and roll your own userspace (thus higher-performance due to the lack of ring transitions) networking layer on top of it. Writing an entire OS is quixotic in this day and age. Using it as a *selling point* is bizarre, like advertising that you smelted your own tin for the solder or made your own ASICs. There's a reason this stuff has been commodified.
Sorry you feel that way but we are more than capable of making an good operating system.
DeleteMaking your own OS is probably why it boots so fast ~2 seconds did you say before?
DeleteThat's insanely fast.
And your codebase is pretty mature, isn't it? Twenty years this year??
DeleteWell yes, in fact we have done tests where a brick was able to shut down ppp cleanly, reboot to new version, and bring up ppp so fast that a one second pong did not even drop a packet.
DeleteTo expand a little. We have been doing this for nearly two decades, and a “normal” o/s would not have run on the first hardware anyway.
DeleteWhile I have my complaints about the Firebrick (configuration so complex I can't figure it out) I'm with RevK on the OS. A product like this does not need everything that a heavyweight OS like Linux can provide, indeed Linux would be a millstone around the neck. I work on embedded processors in a chip. There are multiple cpus with multiple embedded OS's written in house, each specific to what they need to do. The code is related but they are not the same. There is probably no more than 5 man years work in the OS's in total, and that's for three of them on three different architectures. The chip boots in half a second, try doing that on five cpus running Linux. Yes our OS has bugs in it, we found one last month. The previous one was found two years ago.
DeleteI've worked in embedded too --- a simple executive with basic interrupt handling, no threads, no filesystem, and state-machine based logic is not only easy to write but fast, small and secure. (Although it's possible to cock it up --- go google for 'mars pathfinder priority inversion'.) (But I'd also say that VxWorks is pushing the limit of the definition of a simple OS.)
DeleteRevK, I'd be really interested to see a writeup on the FireBrick software, if you have one / feel like writing one...
Linux can boot in 0.07s now if the hardware cooperates or it's a VM without hardware to worry about (see Intel's Clear Containers project for an example). Half a second to userspace is trivial. I can do that on completely random commodity x86 hardware with very little efford. Even my home server, which I haven't tried to optimize boot times for at all since it spends three minutes in POST, boots to userspace in 2s despite having to spend most of that time initializing a dozen-core CPU.
DeleteI'm aware that the FireBrick may not need everything Linux can provide, but what you *cannot* provide yourself is sufficient testing to exercise every combination of code paths in whatever OS you've written. In fact nothing can: the combination is superastronomical, and for any concurrent system is far worse than that. So unless you're formally proving the entire system (which nobody does), your only hope of reliability is being very rigorous and spending vast sums testing, and *still* get something much less well-tested than a commodity OS. And... why do that for the base OS when OSes that have *already had that work done* are in very wide use and available for no cost, and independently verifiable? But, hey, maybe you want to waste vast amounts of money doing something that other people have already done and given away for nothing. Seems like a waste of time and money to me, but so be it.
But the trust issue is not resolvable this way. Honestly, "we wrote our own OS, just trust us that it has no backdoors in it nor bugs that can serve as backdoors" is exactly the sort of thing that reduces my trust in whoever says it, just like "we wrote our own crypto" would and for exactly the same reasons: it's exactly what you'd say if you wanted to quietly introduce a bunch of backdoors. The way to be sure there are no backdoors in something is to make it *publically visible*, not to say "we wrote it all ourselves, it is made in the UK".
I mean, so what if you wrote it all yourselves? That provides the very opposite of grounds for trust! I have no evidence that you didn't accidentally employ someone with every reason to add a backdoor on the sly, and with the code closed and only available to you there's no way anyone other than you can be sure. Heck if you want to be all-out paranoid I have no evidence that you're not *all* a really clever front for a bunch of criminals! All I really know about you is that I pay you money and you give me a very good network service: that provides no guarantee that you're not biding your time waiting to do nefarious things with, say, the software in the FireBrick, which in this scenario you've been quietly adding all sorts of evil backdoors to for years while claiming not to do so. The thing is, if the FireBrick software was open, or based on something which was open, *we would not need any such impossible-to-provide guarantee*: even closed OSes like Windows get their source provided to all sorts of governments specifically so that the governments can audit it (but why should only governments get that sort of guarantee?). I cannot see any reason to keep the source for things like this closed other than dependence on software from a third party with similar requirements (which you say you don't have since you wrote all the code yourselves) or slipping backdoors in.
But perhaps you're simply not thinking about it, and doing it this way because that's the way you always did it and back in the early 90s closed was the default. I'd like to think this is way more likely, but, as noted above, I have no actual *evidence* that you're not criminals pulling a thirty-year-long game, so I have nothing on which to base such a judgement. It would certainly be very sly to argue loudly against backdoors while introducing them yourselves, so if you really are doing this you deserve some sort of Evil Bastard award, a round of applause and running out of the country ;)
[more in next comment]
[continued from last comment]
DeleteI still cannot see why you imagine that this sort of thing increases trust in the FireBrick rather than decreases it. Heck, I'd be a bit worried that my traffic is passing through them now, only it's well-known that the backbones *do* have taps in them, so I don't really care if the FireBricks do as well: I have to act as if my dataflows are being tapped regardless. I *am* now worried that unanticipated bugs in the FireBricks might fubar data in transit, because you simply cannot get the magnitude of software testing at one not very large ISP that (say) Linux has got with its many billions of installations (or, for that matter, Windows, or VxWorks, or *anything* you didn't write yourselves). But maybe you've been very, very lucky or unusually brilliant and written bug-free code, so you don't need the extra reliability that comes from widespread installations! (I don't believe it for a moment and neither should you.)
I appreciate your comments, but there are plenty of potential customers who appreciate what we do, and that we are in control of what we do. I also fully understand the "trust" issue is one that is basically impossible to resolve, even if we "claimed" to use linux. As for the "we wrote our own crypto" the huge issue is designing your own crypto, it is a far lesser issue to code to the existing designed standards. But yes, I do understand what you are saying, I just think that what we are doing avoids more problems than it causes, and we have had decades of experience with this (after decades of experience in embedded systems before that).
Delete(Note: I *do* trust you, but I realise I have no rational grounds for doing so beyond the game-theoretical one that "you get more milk from a cow by milking it", and the way you're acting here actually makes you seem less trustworthy than you did before. Society runs on trust, though, so I trust by default. It's not like I have much in the way of secrets to lose anyway, other than authorization data.)
DeleteThe trust issue is not impossible to resolve: if you used Linux (or any other source-open OS) and opened everything that went onto the FireBrick, *and the FireBrick was reflashable*, nobody would need to trust anything but the hardware itself. Sure, you could still burn a malicious ASIC, but that's a good bit harder...
(And... the history of crypto implementations is rife with people who really really shouldn't have been implementing the damn things. It's *really* easy to bugger them up and not notice. It's really easy to bugger up network stacks and not notice, too, as you doubtless know better than I do!)
Oh, I know how easy it is to mess up network stacks, we are really not new to this, honest!
DeleteNick makes some theroretically valid points, but recent commodity-market vendor history somewhat negates them, and personally I don't see that changing in the foreseeable future.
DeleteE.g. why should anyone trust Intel x86-based products these days? Long-denied wide open management back doors in Intel desktop, laptop, and server chips/chipsets [1], and Intel (Atom) SoCs (in routers etc) that die painfully after a couple of years are just two examples that preceded the latest Intel PR success (Meltdown/Spectre).
Why should anyone trust Linux from a big name these days? Does systemd ring any bells ? Does systemd come from one person, or a corporate? (Or maybe both)? What kind of track record have they got?
I've never done business with AAISP, but the best ISP I have ever been a customer of was Metronet. A handful of very competent very motivated people doing a mostly DIY job. No Redbacks, no Junipers, mostly just DIY software on top of *BSD. It mostly worked fine, till eventually BT Sheffield made them an offer they couldn't refuse.
Conversely, commercial success and a big name does not imply quality or credibility or trustworthiness. Quite often the reverse seems to apply these days.
Intel own WindRiver who own VxWorks, btw. Other commodity embedded OSes may be available, but if you want to eliminate Linux and VxWorks for reasons of "trust", it reduces the options somewhat, especially if flawed logic leads to *BSD being ruled out.
So my experience proves DIY can work, if done properly. My experience also proves that trusting big names is more about purchasing departments and legal departments comfort zones, than it is about a product/supplier's engineering or commercial fitness for purpose.
If we want to look outside the technology market, the same thinking applies. We were supposed to be able to trust RBS (and their auditors, and their regulators, and all the other big names). How well did that work out?
YMMV etc.
[1] https://semiaccurate.com/2017/05/01/remote-security-exploit-2008-intel-platforms/ and others
[2] https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/ and others
@Nick:
DeleteAnother voice here from the "you have valid arguments, but I agree with RevK on this one" camp.
Linux is an absolute mess, probably as much a mess as Windows these days, of heavily backward-compatible code (yet it does a worse job of that than Windows, and only makes up for it by being open source).
I've put CentOS 7 (one of the most stable platforms for a regular x86_64 server) on multiple Intel Atom device and had it just freeze up (anywhere between 10 minutes and 10 hours after booting) due to some bug in the scheduling part of the Linux kernel which nobody seems to understand. Linux is full of weird things like this.
Even if it is proprietary, I would MUCH rather have a small, specialised OS on my router than Linux!
"We control every line of code from scratch and ensure no back doors."
ReplyDeleteProve it :)
Sadly that is not so easy. “Trust us” is probably the best I can do, sorry.
DeleteSo... this is actually worse than *every other OS vendor*, all of which either have their code open already or had it leaked long ago :)
DeleteDo you know the pricing yet?
ReplyDeleteLess than FB2700 and should be published as soon as we are shipping.
DeleteExcellent; it sounds a perfect fit for my needs, as long as it's not out of budget :)
DeleteA blog post would be interesting some time on why you decided to write your own OS. Was it because it was "fun" and satisfying to do so or because you could make it more efficient than an existing OS?
ReplyDeleteIt started as a necessity, a normal o/s was not available for the first hardware we made.
DeleteMay I suggest two ideas for the future: dual PSUs and also some locking mechanism for the power cable(s). Can it connect to IEC ports on PDUs or does it have to be on a 13amp 3 pin plug?
ReplyDeleteWe have dual PSU (and cables that lock) on the larger modes (FB6000). But of course it can connect to a PDU, the cable is standard, and cables from PDUs (three pin IEC) to figure 8 (two pin IEC) exist.
DeleteDoes it support SSH/HTTPS/SNMPv3?
ReplyDeleteFunny you should ask. At present secure off site config would use IPSec tunnel, but TLS for ssh and https are being worked on now. We have not had any call for snmpv3 yet but I’ll look in to that.
DeleteNaive question... I accept that it's sufficient, but is 750Mbps routed traffic good? Why quote megabits rather than packets per second? How would it compare to, say, a cheap linux box with a couple of NICs?
ReplyDeleteMost people would not know what pps they need to be honest. It is what it is, but to be fair we have not tried to optimise the new drivers yet, so just as well the s/w upgrades are free.
DeleteI'm not your target market for a firebrick. I'm a geek, but not an enterprise level geek. My home network is currently all Ubiquiti, including the gateway.
ReplyDeleteIn some ways I would like a firebrick though. I support your stance on a range of issues and the activism you have shown with internet freedoms etc, and if nothing else actually trust you when you say there are no backdoors etc, and have more faith that you will have adhered to standards (both safety and technical) than I do most other companies.
It's obviously overkill for my 350mbit virgin media home connection, but if it could do things other than IPSec, for reasons such as helping me get around crappiness with public WiFi as you've blogged about before, I would probably buy one if it was around the fb2700 price.
IPSec traffic seems commonly blocked, probably because it's obviously a VPN, but SSL/DTLS VPNs and or some sort of iodine vpn-over-dns server would interest me :)
Like I say, I'm not your target audience, so I don't expect you to pay any attention to my desires :)
Why do you say IPSec is commonly blocked? There are perfectly legitimate uses for it, eg. working from home and using IPSec to join the company network in a secure way. Yes this is a VPN, that's the entire point! I have the option of using a VPN to work from home with my current employer. I don't know how it is implemented at the network level as I have neither asked nor checked, but that isn't the point.
DeleteCommonly blocked in certain places. For example the Centre Parcs Wifi last week didn't let me connect to my home IPSec/L2TP VPN but was fine with me connecting to an OpenVPN SSL VPN. I think I remember RevK saying he had issues on most VPNs on one of his cruises, etc.
DeleteOK do Centre Parcs claim to be offering "Internet Service", if so you need to explain that net neutrality laws mean it is illegal for them to block IPsec.
DeleteThe UK has net neutrality laws?
Deletebtw, total number of employers I have ever had since the Internet started who did not block IPsec: 0. (But that's irrelevant, because it's so damn hard to get working that I've always looked at it, screamed, thrown it overboard and gone back to a nice simple SSH tunnel. Sure, if I needed a long-term encrypted reliable link somewhere... well, I'd probably *still* not use IPsec, to be honest. It's just too hard to get working. As various leaks over the past few years have made clear, this was intentional on the part of the NSA folk who added much of that pointless complexity to the protocol...)
The whole EU has them!
DeleteOne thing about IPSec is that it's really easy to block or detect: just match all packets with IP protocol 50 or 51. Contrast ssh or OpenVPN which need slightly smarter packet inspection… which when IPSec came out was relatively expensive.
DeleteWhy on earth would someone block IPSec? We were looking at Center Parcs as a potential for a holiday with some friends, but that would be a deal breaker, as I simply couldn't work if I was there and reliant on their Wi-Fi, and I couldn't be sure I'd get cellular coverage. I've asked their Twitter team if they block IPSec...
DeleteWhether the net neutrality rules apply is an interesting and untested one: given BEREC's quite narrow definition of "public", were I Center Parcs, I'd argue that I provided the Wi-Fi service only to guests of the park, and so am not a "public" provider.
FWIW, I use a "connect-on-demand" IPSec connection on both my phone (iOS) and my laptop (macOS), back through to my FB2700, so it connects to the VPN whenever the device is not on a trusted network.
DeleteWith the upcoming tweaks to the IPSec stuff, I'm sure it will get better still — and Adrian knows I'll keep bugging him about the couple of bits I hope will find their way onto the list of improvements :)
So, certainly at Centre Parcs Longleat, the in-chalet wifi is provided over a DOCSIS network, via a modem and a Ruckus wifi point in each chalet (I found it in the utility room!) I assume this is partly because it's only come about in recent years and meant they could re-use the existing TV distribution network between the chalets to add it.
DeleteIt does in fact give an IP in a Virgin Media range when using it.
It's also pretty crap, I struggled to stream video over it.
IPSec didn't work for me at all, SSL VPNs did. I could possibly have argued with them about the definition of "internet access" but actually I think they only advertised "free wifi", which I guess would be technically accurate even if it was only on to a LAN with no Internet access...
Anyway, in terms of coverage, I had 4G everywhere on Vodafone (I just used this) except in the lower floor of the plaza where the swimming pool was, nothing except small pockets here and there on EE (I have dual SIMs) and my other half got very patchy coverage (although better than EE) on Three. We didn't have anyone on O2.
I would have thought that any WiFi provider would positively encourage IPSec use (or any VPN come to that) - surely any dodgy traffic sent over the VPN is traffic that can't be blamed on them.
ReplyDeleteComplete FireBrick noob, but hardware tinkerer... are there serial/rom debug breakouts on the board? So that i could put my own code on the FB?
ReplyDeleteIf you own the hardware you can do what you like to it, but obviously it is not something we actually offer any help/support for.
DeleteI understand why your own router might be an advantage at your end. Its your business and being able to optimise that is bound to be useful. That's the same reason my own employer wants me to write software for them rather than buying off the shelf. But I don't get why customers would want them as opposed to any other high end consumer router?
ReplyDeleteI have 5 draytek's of various vintages in use for tenants and family members on a mixture of DSL and FTTP on various ISPs. They all talk to each other with VPN's. They're not perfect but I get a pretty consistent experience with them. I've learned their foibles. There are internet forums that can help if I get stuck. A FB would be way too niche for me. Assuming I could even afford 5 of them would you help make it work absolutely anywhere? Actually, you probably would.
Trouble is some of your features like 3G fallback seem to be built on the expectation that there is a FB at both ends. I can't get it to work usefully with my Vigor because the static IP block doesn't move over when DSL is down. This is pretty unfortunate given its a selling point of both yours and theirs!
I've had tickets open twice on this and the conclusions was 'we don't do Draytek's, it works the way we want it to work, you're on your own'.
So, great if you want to roll your own, but don't do that to the exclusion of other customer equipment. You are still very much a premium offering after all.
Btw, thanks for 1TB on BT.
May be worth emailing me re that’s ticket.
Delete