Pointless tests at BT Martlesham

We're having another rather interesting discussion with BT. This time BT plc t/a Openreach.

This is technical, so google the acronyms, sorry. But even if you are not technical you should appreciate the stupidity going on here.

BT have a process for testing modems that work on VDSL / FTTC lines. They even have contract conditions about deploying modems and expect them to pass this test. This does not seem to stop people choosing their own equipment that does not pass the test, so not an issue.

Obviously we want to sell equipment that passes the test if we can. It is getting a bit tedious trying to sort this. But one router we sell, the Zyxel VMG 1312-B10A, was submitted for testing and passed. It is not perfect and we are looking at others, but it is probably the best of the bunch for a consumer route right now, so let's not argue on that for now.

We were really pleased it passed the test, as one of the annoyances with that router is that in bridge mode it does not handle 1508 byte packets which are needed for PPPoE with 1500 byte IP packets. This is complicated, but basically the slightly larger than normal packets allow full size 1500 byte packets to be used on the internet and remove a number of problems with smaller MTUs. The BT modems supplied with VDSL/FTTC (when they supplied modems) did handle these.

Just to do the sums, the Ethernet over VDSL always has one VLAN tag, and could have another or even another. So if you take 12 bytes MAC, 2 bytes type, 4 bytes VLAN, 4 bytes second VLAN, 1508 bytes payload you get 1530 total bytes of Ethernet frame for a 1500 byte IP frame wrapped in 8 bytes of PPPoE header. We are thinking they want to support an extra VLAN tag as they have a VLAN tag as mandatory on the VDSL, so that makes it 1534 bytes total.

The test includes a requirement in SIN498 which states:-

"The modem shall support an Ethernet frame size of between 68 and 1534 bytes. For clarity, this figure includes 4 bytes for the C-VLAN, and excludes bits allocated to pre-amble, Inter-Frame Gap, and Frame Check Sequence at the user network interface (UNI). Support for frame sizes above 1534 bytes (inclusive of C-VLAN) is not guaranteed."

The 68 is the min 64 byte Ethernet with their mandatory 4 byte VLAN tag, and the maximum is the PPPoE maximum baby jumbo frame we worked out above. This spec was designed to allow PPPoE with full 1500 byte IP frames, obviously.

So far so good. I think that is pretty clear. I think, to pass that, I have to be able to send any Ethernet packet size 68 to 1534 bytes and the modem must support it. The Zyxel does not!!! It only handles 1500 byte payload. So cannot handle 1500 byte IP in PPPoE, and requires lower MTU and fragmentation, which has problems.

We know the Zyxel hardware can handle this as a script you can manually run via the CLI after boot makes it work, so this is a silly issue for Zyxel. One small edit to startup scripts and they could comply.

We asked BT why they passed!

Oddly it seems they test with all packet sizes 68 to 1534, and if any packet size works in that range they pass. They have confirmed categorically that if a modem only supported packets of 68 bytes it would pass that part of the test.

So I asked what of a modem that supported more than 1534, and they confirmed that it would, but they only test up to 1534.

Now, I hope that, even if you cannot understand all of the technical aspects here, you can see the logical flaw here.

The number 1534 plays no part in whether a typical VDSL modem passes the test or not! A model supporting only packet sizes up to 1526 passes, as does a modem that supports packet sizes up to 9000.

What is extra odd is the change notes confirm they actually reduced this spec from 1536 to 1534 at one point. They changed a number that has no bearing on the test result.

I can only assume that BT Martlesham have lost the plot and do not know how to do a simple test like this, or understand a simple requirement, or understand Ethernet or IP or PPPoE at all. I thought they were quite technical! Indeed, the test spec (elsewhere in the SIN) talks of checking the maximum supported frame size, and so does not actually test the requirements.

What I did find in the change notes is: "Requirement R.ETH.1 – Text reworded to clarify that multiple MTU sizes can be supported, one of which must be 1534 bytes". Well, "one of which must be 1534 bytes" is pretty clear.

In summary, I cannot take the testing seriously any more. We'll be reviewing modems based on our own tests and not on BTs test spec from now on.

---

This also goes a little to some of OFCOMs work recently, on things like compensation for faults. This spec means that the service we buy from BT, when we got a BT modem (that had the same spec) as part of the service, it does not fit with selling an Internet Access Service. If they supplied modems that happened not to work on Ethernet packet sizes of 142 bytes long, but worked for the rest, it would pass their test and meet the definition of the service. But that is not a valid thing to do when selling an Internet Access Service. If you cannot actually have components from a near monopoly supplier like BT plc that allow you to actually sell an Internet Access Service, how can you possibly have any automatic compensation?