Back doors

Obviously, especially since Snowden, we are all concerned over "back doors" in systems.

But we have had some interesting discussions in the office. Working on FireBrick we make hardware and software from scratch. But even then we are using standard parts, such as processors and Ethernet controllers, and so on.

One of the mind games we play is trying to work out how someone could infiltrate us, using social engineering or technologically or whatever. It is a fun game, but is worth considering, in case we find any defences.

So we pondered, what if the chips we use had back doors? What could those back doors be, and how could they work.

Well, I had two ideas. One was something that tries to pass information to "them", via Ethernet frames. But such a system would be spotted. If not by us during testing, but by millions of other people.

But a simpler idea is something passive - even in a simple Ethernet controller. These things have access to the memory of the system via the bus and DMA and so on. They need this to send and receive legitimate packets.

If I wanted to implant a back door, I would make an Ethernet controller able to respond to a specially crafted packet. Instead of passing that to the processor as normal, it would take some action and send a reply packet. The action could simply be to allow reading or writing of system memory using the same DMA and memory access needed to send and receive normal packets.

The upshot would be that nothing would be detectable unless targeted.

But if targeted, the packets would look like normal IP packets. The payloads could even be scrambled or encrypted in some way. It could be used to attack anything that is accessible on the Internet and provide a way to access the running memory of the system remotely.

This could allow access to private keys for encryption and allow patching of code live to add proper back doors.

Now, this could apply to an Ethernet controller chip, or even a library part included in a custom logic gate array. It could be in an Ethernet card or whatever. The back door itself could be tiny in terms of silicon if all it does is read and write memory in response to some simple packet. Even people making their own silicon could find they have a back door!

The only issue is the reputation of the manufacturer if caught out... Is that enough to protect us? If some large company making such devices caved in to pressure? What if a few key employees caved in to a bribe. Scary?

Update: We have checked the coding on the FireBricks and the memory mapping does mean the Ethernet controller can only access the packet buffer memory and not general RAM or Flash which may otherwise hold keys, etc. So we are on top of this risk already to some extent :-)

Manually keyed IPsec

My previous post explained a bit about IPsec. We have released it in phases, with manual keying first, then IKEv2 key negotiation, and now EAP with certificates and road warrior stuff.

As part of the work on this release we considered actually removing manual keying as a feature. Whilst it was a useful first stage of the release of the initial version, it is not as secure because the key itself never changes, so is more vulnerable to decoding intercepted traffic eventually. With IKEv2 the key regularly changes making the job a lot harder. We released IKEv2 18 months ago, soon after the initial manual keying release.

The latest change is not primarily about security but convenience. The EAP changes allow identity to be agreed using certificates and usernames and passwords, but the key exchange and updates are managed by IKEv2. EAP makes it a lot easier for mobile phones to connect in to a FireBrick, and get an IP allocation and so on.

So, removing the support for manual keying was a consideration. We looked at other devices and concluded that manual keying was not that commonly supported and that key exchange was more common. IKEv2 is being added to devices more recently, with some devices having supported IKEv1 (which we do not). But manual keying was pretty rare. Indeed, we did not find any examples.

We decided it was going a bit far to actually remove the feature just yet, but managed to leave a problem where the config from an old version using manual keying would not work on the newest release. This, in itself, was a mistake. Having decided to keep support for manual keying, at least for now, we should have made the config compatible (moving the old format to the new one automatically). But the fact we assumed nobody was using manual keying kind of put that on the bottom of the list, and it was left out.

Sadly, that was a bad call! Sorry.

It seems that we have a wiki that explains setting up FireBrick to FireBrick IPsec using manual keying, and even A&A staff were using that until recently. The upshot is that code updates broke some IPsec tunnels. The wiki should have been updated 18 months ago!

So, we have stopped the roll out of new code today, and we should have a new release tomorrow morning. It will automatically map the old config to the new. It seems we only had a hand full of people affected, and support staff have been working to ensure that they are configured correctly.

The other good news is that anyone upgraded today, and using IPsec manual keying, and that has not touched their config, will be updated tomorrow with the config change to make it work again.

But having realised that some people are using manual keying, we need to find a way to try and get people to upgrade their config to IKEv2 with a pre-shared secret. This is not just because we'd like to remove manual keying, but because it is not as secure. The problem is how we know who is using IPsec with manual keying?

Now, we have some mailing lists, but we emailed asking if anyone was using manual keying as part of the research we did in this release, and had no replies. The problem is a lot of people actually using IPsec have had someone else help them set it up, and have no idea.

So, we have a plan. We are very careful about the way in which a FireBrick phones home. It is a security product and people are understandably concerned about security. We have no back-doors in the product. The closest to phoning home is the software and capability update check, based on a DNS lookup, that allows us to do upgrades - but people can control that in the config. The only other thing we have is the default fb-support log target which is there primarily to email us if a FireBrick crashes. All of this is under the control of the person configuring the FireBrick and can be turned off. Though we don't recommend it as crash logs are useful to help fix any problems, and software updates are important. The key thing here is that we are not secretive about this and we give people a choice.

So, the plan is that the release tomorrow will not only update any IPsec manual keying config, but also log the fact to the fb-support log, which means we'll get an email. It may mean a dozen or so FireBricks email us, and we can then contact people to help them with a config update to use IKEv2 pre-shared secrets instead. If we can be reasonably sure that nobody is using manually keyed IPsec we can then look to remove it from future releases.

This should give us a good mechanism for managing such things in the future. Ultimately, whilst this was a bad call by me on this, even if the number of people affected is tiny, we learn from such mistakes and aim to improve the product. Ironically, I suspect this blog will make more people aware of this issue than the mailing lists, which is a shame. If you use a FireBrick, please do sign up.

Update, we are releasing 1.36.002 today (29th) which upgrades the config from older format to the new format for manually keyed IPsec, and logging if manual keying is being used.

IPsec

We have spent literally years working on IPsec in the FireBrick. It is a complex project but it is finally getting to an end, well, sort of - at least a major milestone and release.

IPsec is very much seen as an industry standard way to create Virtual Private Network (VPN) links, both point to point between offices, and "road warrior" roaming from mobile networks in to office networks.

The problem is that IPsec is a lot of layers, a lot of standards, and far from simple. There are the low level encryption and hashing algorithms. These take a lot of work to implement and test. They also need a lot of low level maths functions coding as well. There are then layers and layers of protocol on top.

For a long time the FireBrick has supported manual keying - that means the key (typically entered as a long hex string) is entered both ends and is fixed. This was the first layer and meant having all of the basic algorithms working. We recommend that anyone still using manual keying between FireBricks changes to using IKEv2 and a pre-shared key.

We then added IKEv2 (18 months ago), which is a key exchange protocol. This allowed keys to be negotiated dynamically rather than being fixed in the config at each end. This is a big improvement on the manual keying, and allows the key exchange based on a simple shared pass phrase. We did not implement IKEv1 as it was not quite such a clean standard and many devices are now doing IKEv2 (even apple).

This final stage involves EAP, which is complicated by the fact that certificates are used to authenticate the server (FireBrick) end. This has meant implementing the whole system for managing checking and signing using certificates and keys. The client can then authenticate using a simple username and password. It add to the fun, iPhones do not allow a simple manually entered config, but a profile file that is loaded. In some ways this helps as end users can just click on it from an email, but it makes it more complex for the sysadmin to set up.

The upshot of this that iPhones and Androids can connect in to an office LAN securely.

Now, when I say "we", I mean pretty much "Cliff". He has spent day and night working on this. The work we have done gives a good foundation using our own code (so no NSA/GCHQ back doors) and allows a lot more work to be done in encryption in the FireBrick. The next generation of hardware we are working on even has a true random number generator built in as well as some options for hardware encryption accelleration. We know people have asked about OpenVPN, and we are looking in to this as well.

Even so, the IPsec setup is still complex, and I have made a cheat sheet. But hey, if it was not complicated then my friends would not be able to sell consultancy :-)

The latest code is in beta now, and should be a factory release shortly. We suspect there will be a few more bits of work down the line on this and new releases in due course, but now we finally have it working with common mobile devices we can start working on some of the other new features in the FireBrick code. So, well done Cliff.

What is a broadband fault?

Exterminate the orc?

Continuing a theme here, but my mate Mike came over. One of the games he is quite good at is playing devil's advocate, rather well, and he actually made some really good arguments on the issue of SFI charges and faults. I'd like to thank him for that, oh, and for the dalek he brought over to surprise us!

The crux of his point is that the very definition of a broadband service is that it is rate adaptive and so it is a service that will try and get the best performance on the voice grade copper pair to which it is attached. It is possible for the characteristics of a voice grade copper pair to be so bad (very long line) that it cannot even do broadband, and that would be tough. That basically the performance of the broadband line is kind of whatever it is. Indeed, a comment on my previous post about TalkTalk was raising this very point.

He has a very good point, and it has allowed me to hone the arguments I can use against such points with the likes of BT and TalkTalk.

Yes, the broadband is what it is, "as well as it can do" on the voice grade copper pair to which it is attached. We do accept that to some extent. There are caveats - in that the copper pair will have a forecast speed, and if the line is way out of that, then that probably means something is wrong. But in general, you get what you get. However, this relates to the rate of the broadband, as it is rate adaptive.

I would argue that there is a huge difference between a line that is a low rate on a line because it is all the line can do, even if that means the rate has reduced a bit over time due to increases in cross talk from other lines and so on, and an actual fault.

The issue comes down to how you define a fault in the broadband. This is different to how you define a fault in the voice grade copper pair.

One of the key aspects in defining a fault - perhaps the key aspect - is that a fault can be fixed. It is a condition that is the result of some aspect of the copper pair which is not as good as it could be giving the length and routing of the pair. Faults could be down to poor joints, corrosion, water ingress, poor insulation, electrical contacts, degraded materials. It could be fixed by using a different pair, or repairing joints or equipment.

So how do you spot a fault? Well, BT plc t/a BT Wholesale actually defined one way of doing this. They monitor the line in the first 10 days and find the maximum stable rate for the line, and set a fault threshold rate (FTR) for the line based on that (with a percentage taken off). This defines a sync speed below which the line is considered to have a broadband fault. That is excellent as it gives a crystal clear metric which we can all agree defines a fault. TalkTalk don't do this, but it would not be hard for us to do the same based on sync speed history and agree a metric with them.

Another way to measure a fault would be to consider that the line loses sync lots of times. To be honest, once a day is too often. Normal lines stay in sync indefinitely and would only lose sync do to some external interference, or a power blip or some such. So frequent loss of sync should be a fault. Bear in mind, even a long and highly lossy line should stay in sync, albeit at a slower speed.

We also have metrics in terms of packet loss. Whilst we measure latency, that is not normally a clue to any sort of line/sync issue and more likely to be backhaul congestion (or a router moving enough traffic to build up a queue). I think packet loss (when no traffic) is a really good indicator of a fault. Also, if the line has any internal metrics (OAM frames, Header errors, FEC errors) that is an indication of a fault.

Indeed, the DSLAM can almost certainly report if there are lots of bit swaps, problem frequency bins, FEC errors, HEC or other error stats that indicate a fault, as opposed to simply being a poor/long line.

Unfortunately, neither BT nor TalkTalk define metrics for packet loss, or line errors or re-syncs as a fault metric. I think they should.

Fortunately, when there is a fault, the levels are pretty clear cut. It is rare for a line to have a "tiny bit of loss", though not impossible.

In practice, the fact that the broadband has a fault of some sort, based on our monitoring, or the DSLAM stats, is not itself usually the issue. Usually we can agree that there is an issue with the broadband. We can also usually agree that the copper pair meets the voice spec, when it does.

One of the key things that tells us we have a fault is when a line that used to just work fine (at whatever speed it had) now has problems which it did not have before by any of those metrics. That suggests it can be fixed and made to be back the way it used to be by some means. We know what is possible at that point.

I think we just need to pin down ground rules with providers for what is and is not a broadband fault, and pin down that they are responsible for broadband faults, not us.

So let's start the discussions with them, and try to make things better. Well done TalkTalk for being the ones wanting to talk at this stage. Well, either that or we send round the exterminator!

Did not fit through the door

Race to the bottom

Update: TalkTalk really are trying - and this looks like a case of a discussion point on this one line rather than a more general policy, so stand down the panic for now... Let's talk to them!

Latest from TalkTalk - if the broadband service they sell us gets a fault, their side of the agreed demarcation point (e.g. in the line itself), they will not even try and fix it, but will decide that they can no longer provide the service.

So get a fault - fix by ceasing the service.

I guess that is one way to stitch up your customer!

Well done TalkTalk - you are sinking lower than BT Wholesale now.

Update: We may be making some progress on this, so no need for everyone to ask us to move them back to BT just yet, but thanks to those that did ask. I do hope that we will soon have more sensible ways of working with both providers.

The plot thickens

To SFI or not to SFI?

BT plc t/a BT Wholesale have stated that "SFI2 is an Openreach service which is made available to BT Wholesale customers & charged for on a modular basis." and "The SFI2 visit simply checks whether a line is working within the specification of SIN 349." This is, of course, the basis of my various rants on the matter, and that by that definition there is no way to get a broadband fault fixed.

However, BT plc t/a Openreach, who (from the above) actually provide the SFI2 service state that "SFI2 is a chargeable investigation product that attempts to identify and resolve Digital Subscriber Line (DSL) Service affecting problems." and it goes on to explain that this service is used when the line "is apparently working within the LLU contractual specification of SIN349".

It goes on to explain the modules available, checking at the exchange, and checking the network, etc. It goes on to list all of the steps done by the engineer in the initial (base) module to identify the cause of the problem, such as checking the modem is connected and shows sync. It does say that the engineer does a pair quality test, and if that fails, he will work on basis of a line fault first, and then go back to trying to resolve any remaining broadband issues.

This is, basically, the way SFI2 engineers used to be defined by BT plc t/a BT Wholesale. So why have BT plc t/a BT Wholesale changed their definition of this service, whilst now claiming it is simply a service provided by BT plc t/a Openreach and offered to us. And why are they trying to charge us for it? It is defined as service they can buy to fix the service they sell us (broadband) and even defined in such a way that it should only by used where the line meets SIN349 already, which means BT plc t/a BT Wholesales charging (when it meets SIN349) would always charge us. It would be a mistake for an SFI2 visit to not be chargeable as it should not have been requested in the first place if the line does not meet SIN349.

One of them, either BT plc t/a BT Wholesale, or BT plc t/a Openreach, must surely be lying to us? And the only possible reason we can think of for doing so is to charge us money, making such a lie in to criminal fraud, in my opinion.

We've asked questions about the apparent difference between the two descriptions of the service, and await answers. However, the previous email on this has been weeks with no reply.

For now, we have to work on BT plc t/a BT Wholesale's statements as to this optional service which we would never want or need, and ask them to actually fix the broadband.

It is worth mentioning that this may impact how we deal with TalkTalk slightly, as they are not making the same claims about what the SFI2 service provides. They are, also, responding to us and asking to discuss how we can work together to come up with something better. Well done Talk Talk.

Trying to do the right thing

These new migration rules for broadband are a bloody nightmare. We have already had to contact customers for explicit consent to email the notices (thanks to all of those that have confirmed so far). But the next challenge I am facing is the contents of that Notice of Transfer.

One of the things we have to include is details of the amount of the Early Termination Charge due at the expected Migration Date.

This may sound simple, but it is far from it. I'll explain some of my problems here, and what I am thinking of doing.

30 days notice

Our units based tariffs operate on a simple 30 days notice basis. This is simple as the current system of migration code (MAC) is valid for 30 days, so asking for one is giving us 30 days notice and we invoice (or credit) so you are paying up to 30 days time when you request the MAC.

With the new system the first we hear is when we get a notice of transfer, which has a 10 day lead time. Now. if we stick to 30 days notice, does that mean there is a 20 day "early termination charge"? Well no - it is not an early termination charge is it? It applies after one month or after 10 years of service, it is not a charge for terminating "early" in any way.

But we also run in to an issue that someone could "give us notice" on the 1st that they will be migrating at the end of the month, and then start a migrate on 20th (with 10 days lead time) and rightly expect not to pay any extra as they gave notice 30 days before. We are simply not geared up for that, so I expect we will change the logic so there is no "30 days notice" any more, and just a final bill adjusting to the termination date. We may make the units tariff have a 1 month min term in that case.

Minimum term services

This should, surely be simple? We have Home::1 with 6 months minimum term, Office::1 and FTTC with 12 months minimum term. So when you migrate there is the charge for the remainder of the minimum term.

The complication here is that if the service has already been billed to the end of the month, or quarter, is the "early termination charge" the value of the final invoice you get, extending that to the minimum term, or is it the charge from the migration date to the end of the minimum term which includes some you have already been billed?

Also, if you ask to migrate on say the 28th of the month, you get a regular month's invoice on 1st, and then migrate on say the 7th of next month, should the bill you have not yet had when you start the migration be considered part of the early termination charge?

My current plan is to specify that you are migrating on date A, and the min term is the later date B, and so the charges from A to B are X months at £Y making an "early termination charge" of £Z, and then noting that you have already been charged up to date C and so some of that charge has already been invoiced. Even that is complicated and may need a further note that you will get your regular invoice for services from date D as normal which may be before date A. How the hell we word this in a way that is clear to the end user is not easy.

This is with what I consider a "simple" system that is just a minimum term.

Combined services

This is where is gets really complicated. We have PSTN lines / copper pairs provided only for broadband service, so you have two services. We could get migration notice for one or the other, or both.

For a start, if we get a notice for the broadband we have to warn the end use that we will cease the copper pair when the leave, killing their migrated broadband if they don't also arrange to migrate the copper pair.

But the "early termination" is complex. The amount we'll charge for minimum term will be for both copper pair and broadband to the end of the min term. So do we tell people that when we get the notice for the broadband? We'll have to explain that both services will stop early.

Of course migrating the copper pair could mean keeping the broadband with us, so maybe that just needs to give notice for the copper pair min term. Though we could possibly not have a minimum term on that aspect of the service - that would be one option. It complicates matters for customers when some parts of the service have a minimum term and some do not.

Multiple lines

Of course, it gets even more complex with multiple line services. Our Office::1 service is two lines minimum. We do not do a single line option, or rather the price for a single line service is same as two lines. So if migrating one line, there is no early termination charge - you just keep paying for the service at full price on the remaining one line service. But do we say that (no termination charge, service continues on one line) for each notice we get? Or do we some how work out that we have had notices for the other line as well or the PSTN for the other lines?

Office::1 requires our PSTN lines, so is a migrate for PSTN to be treated as ceasing that line for broadband too? Maybe migration of any PSTN or broadband on Office::1 should be taken as notice to cease the whole Office::1 package? That may be the simplest approach here.

We have similar issues with units tariffs where the units part applies to the lines collectively. Cancel one line of a set and the units charges carry on - cancelling the final line, and the units charges stop or have some early termination charge. Now this is where making units tariff simply bill/credit to the migrate date and not have an early termination charge is simpler, but making it one month minimum term would still leave a possibility of complications. Maybe simpler for the units not to be refunded - i.e. they are charged in advance and applied in full to the period, full stop. After all, we already have complications if someone has a month of units and uses all of that data and leaves mid month. But is that fair?

I think, simplest option will be, PSTN/copper pair lines have no minimum term, and can be migrated away independently of the broadband (except Office::1). That solves the migrates on that side. I think that any multiple line package with a migrate of broadband should be treated as ceasing the whole multiple line service. That should make the notice clearer and can explain the costs for minimum term on the whole service.

No refunds?

One system we used to have when we first started was a no refunds system. It meant you are billed in advance each month. If you leave by any means during that month, you are not refunded for what you have been billed. This is a much simpler system that 30 days notice, and makes a lot of this easy. It is not an early termination charge, so nothing special needs to be included in the NoT. It would work possible for units billing and services beyond the minimum term, but those within minimum term could still be complex. It creates issues for people paying quarterly. It creates issues with all ceases and migrates being last day of the month. Whilst it is simpler in many ways I suspect it is not that viable.

Trying to be fair...

I wonder how else we can do this and be fair to customers. We do have minimum terms we pay carriers for things like FTTC and even one month on ADSL. The Home::1 and Office::1 are priced on the basis that we have the customer for a minimum term as we make a loss on the set up charges. I suspect we have to change some of the logic for what we do in terms of notice and invoices/credits on leaving. We also have to consider how these changes relate to people who cease services rather than migrate away. We could go for simple 10 day notice for cease to align with the migrate system in the way it works, perhaps.

Other ideas?

One idea was to simply have a fixed price early termination charge on the minimum term services. This would fit the OFCOM model perfectly, but it means OFCOM dictating the terms on which we do business, which is getting silly.



Why the hell do OFCOM have to meddle?!?!

How do you know your core routers have no back doors?

Taking security seriously

Spent an hour talking to a nice journalist - it happens...

He wanted to know about privacy and how A&A do things. I'll leave him to make an article on that - I have done many before.

But I did like the question on equipment - how do we know that the equipment in our network does not have back doors, do we audit the code somehow or what?

I explained, "we build them", as in PCB design, code from first line of boot, and operating system and Ethernet drivers, and TCP stack and BGP and IPsec and LNS and everything with hardware made in UK (Newbury), so we know... I even explained that the team is small enough that we know we have no NSA/GCHQ moles. He asked if I was one, and I re-assured him that I am not! Good question!

As usual, I think it blew his mind slightly. That, or it made us sound like a bunch of tin foil hat wearers. But FireBrick does take things seriously...

New TLDs

The whole thing with the new TLDs (Top Level Domains) is quite fun - so many new domains means some fun host names.

I have to be careful and not use my outdated knowledge of how things work to say "that is wrong", so when someone had and issue today and I saw their VoIP registration was to a host called "telephone.exchange" I had to take a step back and double check that...

Yes, that is a real host name :-

;; ANSWER SECTION:

telephone.exchange. 2973 IN A 81.2.101.128

Further in to the rabbit hole

OK, BT plc t/a BT Wholesale try to charge us for SFI2 engineers to fix a broadband fault (if the line meets SIN349)...

But Talk Talk are going further.

TalkTalk Communications t/a TalkTalk Business are charging us for exchange work. Re-jumpering the line in the exchange.

You really do not get much more "their side of the demarcation point (master socket)" than that!

We will never pay any broadband provider any extra to fix the broadband service they already sell us. That is about as simple as you can get.

One of our guys (Stuart) actually dug out and quoted sections from the Fraud Act in explaining this. Well done Stuart. Yes, it may even be CRIMINAL FRAUD for people to try and extort money from us to fix the service we already buy from them, in my opinion.

We'll try and keep it breach of contract and not involve the police at this stage, but it is an interesting option...

Emailing people asking if it is OK to email them?!

No, not spam related this time, but I have emailed A&A customers that have lines from us asking them to confirm they are happy for us to email them.

I have not gone insane, honest. OFCOM have made some rules, and to comply with them, we have to have explicit consent to email notices to customers. This is in spite of the fact we make it clear in our terms and customers are well aware that we email invoices, DD notices, and pretty much everything else.

The only mistake I made is that I should have PGP signed the email! Sorry about that.

I have tried to make it as simple as possible - one link from the email and a confirm button. I'd appreciate it if people could follow it and confirm.

We'll email people again if they don't confirm - sorry - but if you don't like it please do complain to OFCOM for being so "creative" with these new rules.

Update: I have resent the email to people that have not yet confirmed, PGP signed. Thank you for the comments on this. So far we have two thirds confirmed, thank you.

It is rare to sack a customer

But when someone replies to the automated ticket response with this, I think we have to.

Think yoursleves lucky we're hundreds of miles apart.  

After the hours of aggravation you've just caused me nothing would give me greater pleasure than to pop round to your offices and sort you out, man to useless parasite.  

Some of us create and produce.  Others suck our blood like leeches.  Well go suck yourselves.  

Trust me - I'm a journalist - and your name will be sh*t henceforth.

Totally unjustified, obviously - looks like he may have a line issue, but looking at it, it is probably caused by an unfiltered extension connected to a sky box or similar. Something we'd be happy to help him with if he actually reported a fault. I cannot have customers threatening to "sort you out" or calling my staff "parasites". It is simply not acceptable, whatever issue they have with their line.

Update: Thank you Len for this classic comment on your email to us:-

Ps I'm not a journalist and your name will be praised henceforth :))

Day in court

So, suing a spammer today, and to be clear, he did send an unsolicited marketing email to me, an individual subscriber - sadly I lost the case. But as ever, it works out a cheap lesson in how the civil justice system works.

Neil Brown was kind enough to observe and make detailed notes, here. Thank you. It is worth noting that the judge was very happy that this was a public hearing and so there were observers allowed. Indeed, the defendant asked his wife to come in when he realised.

There were a couple of interesting procedural points which are worth noting. For a start, even though the defendant had refused an arbitration call, and even though the judge agreed that is normally considered in costs, the judge award expenses against me!

As for the expenses, it seems it is very worth while bringing all your receipts to the hearing, including, it seems, the cost of recorded delivery postage for the defence and paperwork to the court and other party. I had no idea you could claim such things, but the judge was happy to pay documented expenses, and even 45p a mile for travel. So I'll be doing that in future.

Sadly the point of whether I am an individual subscriber was not addressed, which is a shame. The judge went straight for the idea that I had no evidence of damages and hence that I have no claim. She also said that you cannot claim for distress.

This is a worrying view, as it basically makes regulation 30 pointless in almost all cases.

The good news, of course, is that such cases are not a precedent, and the next judge may well have a different view.

So what to do next...

1. I need a nice clear summary of the recent cases, which do set a precedent, which make it clear that you can claim for distress in cases like this. Had I got this with me I could have countered the assertion that the judge made. There have been several cases where traditional damages were indeed nominal (and £1 awarded) but distress was substantially more (hundreds of pounds).
2. I need to look in to how the value of distress is normally calculated, and then show those calculations for the basis of my next claim. This may, of course, mean I end up claiming more.

Of course, once we have this, I can post details on my blog here for anyone else to use (not legal advice, obviously).

Oh, and obviously, I should report this incident of spam to the ICO and request that they exercise their enforcement function. Sadly they will do nothing as usual.

BT: We'll fix a fault in 40 clock hours, unless it is because we did not invest in the network...

OK, that is my paraphrasing of the situation and not a direct quote, but what it seems to me is happening. I'll be delighted if our BT account manager has a better answer for us, and I'll be happy to publish that.

BT have an agreement to try and fix a broadband fault with 40 clock hours. Often they try to work to this.

But when their own back-haul network is, it seems, not fit for purpose, they'll not even try. We are being told "June" as when they plan to fix this.

It is a clear 21CN back-haul issue, latency in the evenings and even worse on Sunday. A failure of network planning and investment, even though we are paying by the Mb/s and so this actually loses them possible revenue!

As a BT shareholder I am disgusted.

21CN-ACC-ALN1-MR-CHO

Update: We have reported a fault, and as the line tests call come back fine BT have said we need to book an engineer to go and do the same line tests again, for a fee (an SFI2 engineer). We are explaining that the line is fine and hence such an engineer is pointless. I'll update on how far we get.

Update: We have noticed a slight change in attitude, in that even though the automation insists an SFI is the only option, the staff on echat are prepared to pass back to diagnostics when we point out that the line meets SIN349. The problem is that they then send it back as no fault found, and suggest an SFI again. We'll see how it goes.

Update: BT are being unusually helpful and trying to work with us to solve this issue and the whole SFI issue... Can't go in to details, but a surreal call with BT where I was having trouble believing that they were trying to be so pragmatic and helpful. Thanks to our account manager!

Update: I was dreaming - the main issue of SFIs is not yet something we have any progress on - I knew it was unbelievable. Oh well.

My dead mother-in-law

Dear sky,

She has been dead for two years, and you were told.

Please feel free to waste your time and money trying to sue her.

Arses!

Even though she was only my mother-in-law, even I find this sort of harassment distressing. I only have this because I was the contact for email as she did not have email. Not amused.

Man cave

As I say, it is going nowhere, but it is starting to get warmer. This makes my little study a pain.

So I have moved in a minimal set up in to the garage that has the aircon installed. I am glad I did that first! Just the bare essentials, until I can get some actual work done...

Photographing kids

Lewis having fun

My wife and I took our two grandchildren to Hobbledown farm this week. It is set up very well for young children, and they both had a great time. They were totally worn out by the end of the day (and so was I, by lunch time). There was one small incident where the cafe served toast on mouldy bread and then had a "couldn't give a shit" attitude, but the site management were horrified and gave us a ticket to come back - for which Sandra has vowed to bring a picnic! However, it was a fun day.

As I do not get the chance to have a day out with the grandchildren often, I took my camera.

Now, I know that taking photographs is not a crime, and that (in general) taking photographs in public, of the public, and even of children, is not illegal. I was, however, keen to avoid any issues with paranoid parents. As private land they can make rules on such things as a condition of entry, but seem to have no restrictions, and even seem to encourage photos and video in competitions, which is good.

I suspect that I am the one being paranoid here. I get the impression from news stories that the public, and some of those in authority, think of photographers as some sort of terrorists or pedophiles, and to be treated with suspicion. It is a tad ironic that I am far more concerned about backlash against my using a camera than I am about either terrorists or pedophiles ever being an issue for any of my family.

I had no hassle. I saw one other parent with a good camera (well, a Nikon, but you get the idea :-) ).

Bobby taking pictures on a DS

Even so, it is a tad ironic (am I using that word correctly here?) that if parents are concerned over some random person taking pictures of their kids, they might look at those of us with a good camera. With a typical phone camera you will have a wide angle shot with a small aperture (especially as it wass sunny) and get loads of other kids in sharp focus. With a good camera I think I managed to only take shots which have no other children clearly visible (either zoomed in to one subject or shallow depth of field to blur background).

One possible factor was my jacket. I have a high visibility jacket which I use when cycling, and I was mistaken for someone "official" a few times (e.g. asking directions). That said, I did wonder what one would have printed on a hi-vi jacket to warrant taking photographs and the best I could come up with was "FORENSICS", but my wife won't let me put that on my jacket, sadly. It almost certainly helps that I was not seeming to be taking pictures of random kids.

So, maybe people are not as paranoid about photography as I thought. I hope that is the case.

Hashes

I have posted a bit on passwords, and I thought it only fair to try and explain some of the issues to my non-technical friends.

I am going to try and explain what a hash is. This is going to be tricky without talking about mathematics, but I'll try.

A hash is a function that... Ah, OK, I may have to explain the idea of a function first.

A function is normally a process, a sequence of steps that you can take, to turn one thing in to another. In maths, you can have a simple function like multiply by two, and you can use that to turn 3 in to 6, and 4 in to 8, and so on.

OK, now, a hash, is a function, but it is a complicated one.

When you look at a function like multiply by two you can work out that there is an inverse function. That is, a function that does the opposite. In that case it is a divide by two, which turns 6 in to 3, and 8 in to 4, and so on, doing the opposite of the first function.

The idea of a hash function, in this case, is it takes an arbitrarily long string, such as a password, and makes it in to a number, a big number. In fact, such a big number that it is usually written as a long string of hexadecimal digits. But this should not really put you off, it is still just a number.

There are lots of different types of hash functions that can exist, and there are different reasons why you might have them, and different algorithms that make sense to meet those reasons. Here are a few examples. However, mostly, a hash function should be one you cannot do backwards (i.e. cannot make the inverse function).

1. A hash for distribution

Let's say you want to put things in different bins, and you want a way to decide which bin to put them in. We'll say we are talking about words here still (like password), but we want to turn the word in to a number of which bin to put the word in. We don't care which bin we use, but we want to always put the same word in the same bin. OK, that sounds convoluted, but trust me, this is sometimes a thing you want to do with computers!

In this case the hash can be very simple - you could turn each letter in to a number and add them all up, for example. That may not be the best way, but it is a way that will work for this type of scenario. We probably want the hash to be quick and easy to work out, and we are not too worried about how easily it is for someone who guess a word from the hash later or to try and make an inverse function that works out what the word was from the answer. There are several good functions for this, and what makes them good is that they are fast and the create a reasonably even distribution of answers for different inputs.

2. A hash for encryption

Some times you want a hash that is turning a long bit of text or data in to a single (long) number in such a way that it is almost impossible to reverse the process. I.e. there is pretty much no way to find the text that makes the answer to the hash function. In fact, you want to make it so that pretty much any text you use will come up with a different answer with very little chance that two bits of text could ever come up with the same answer.

This is quite important for lots of things in computers, as the hash can be treated as a substitute for the original text. If I know the number and I have a way of verifying that number is correct, I can assume that the text that I later check matches that number is correct.

For example, imagine it is a lot of work to send a big file to you, and at the end you want to check you have the right file and it has not been corrupted, either by accident or deliberately. Well, you could make a hash, and so could I, and we could call each other, and I could tell you the hash and you can check it is right. I don't have to read all of the text over the phone to you, just the hash.

The requirements here are that the hash is quick and easy for a computer to compute, but that the chance of two texts ever matching the same hash is very slim, and the chance of a slightly modified version of a text matching the same hash is pretty much zero, and the chance of someone being able to contrive a text that matches is pretty much zero. You need the hash to be reliably a substitute for the original text for checking.

3. A hash for passwords

The requirements for a hash for passwords are very similar to a hash for encryption, and almost all of the issues of being unable to reverse the hash or find a password that makes the same hash, are the same.

There are, however, a couple of key differences.

For a start, passwords are shorter, so a much more complex algorithm could be used which might be totally infeasible for hashing large files.

But also, you want to make the hash slow and complicated (e.g. use lots of computing power and memory). The reason being that passwords are guessable. E.g. using every word in a dictionary and lots of digits and other tricks people use to make likely passwords, and check them.

If the hash is very quick to work out, a computer can work through a lot of guesses of possible passwords and try them out to brute force the answer and find a password to match.

Indeed, as someone reported on my other blog posts, some graphics cards are able to do around 5 billion SHA1 hashes a second! SHA1 is a cryptographic hash, so designed to be quick to work out by a computer.

The trick for a password hash is to be as secure as a cryptographic hash but also hard to work out in a way that means hackers cannot try a lot of hashes per second on simple cheap hardware.

I hope that makes some sense...

Passwords

I posted on this a while ago and was quite impressed with some of the replies.

We are taking on board many of the comments, and have this week been working on the some changes. The status posts go in to some detail, but I thought I would explain more.

Firstly, there is more than can, and will be done over time, but as with all security it is all about having the right level of inconvenience for the risk.

So what is happening?

Well, for a start, we have used hashes for storing passwords in various places for some time. This means we do not know the password, but we can check it. We make the same hash from the attempted password and check against the hash we have stored.

One of the improvements that has been made is salted hashes where, in addition to the password, a salt is used (random data) when making the hash. This means two people with the same password do not have the same hash, and means you cannot make a huge dictionary of common passwords and combinations of dictionary words with pre-calculated hashes (rainbow tables).

However, sadly, some systems work in such a way that we have to know the "plain text" password. This applies to CHAP (Challenge Handshake Authentication Protocol) and similar systems used for VoIP. To check the password we have to know it. However, the systems that allow control and management and even visibility of such passwords can be locked down under systems that can hash the password.

This is the main change we are making... Our "control pages" (called "clueless") were originally for DSL lines, and that was it. We needed plain text passwords for the CHAP login, and that was the only password.

However, over time, things have become more complex. We now allow control of VoIP logins which can be used to make lots of calls and run up bills, using the control pages login. We also allow visibility of call logs. So we need to beef up security a bit.

Salted hashes where we can.

The first step is to ensure salted hashes are used, not only on our accounts system but also on the control pages. We are changing them so the control pages login is a salted hash, and the line login is a separate line password. The latter is just for DSL/L2TP/SIM login and is often ignored if we have a valid login and circuit ID anyway.

These changes will soon be visible, allowing DSL passwords on lines to be seen/changed, but passwords on the login pages to be set using a password change process with https and no plain text passwords in emails and no way for staff, or anyone else, to see the password.

Making a system that allows improvement.

Doing this has meant we have the opportunity to make new password generation and checking code in common libraries used over many systems. These functions will check a salted hash, but they also have the ability to report that the correct login used an old hash of some sort and provide a new hash. The systems allowing logins then update the hash on the next login.

This allows us to improve things in the future.

For today we are using a salted SHA1. This is not ideal as SHA1 is considered a bit quick and efficient for a password hash, but it is a good starting point. There is a competition for password hashing algorithms running now, and the plan is to move to a new best practice password hash in due course. As it happens, FireBricks are moving to heavily salted SHA256 for now.

Having a system library to handle this means that when we upgrade, it will happen automatically. Using the new hash for all new passwords, and updating the hash on existing logins when used.

Of course, ultimately, these libraries may allow OTP and two factor logic to be deployed where sensible.

Creating passwords

Part of the library is a single place to manage the creation of passwords. We now have a function allowing a password to be generated with specified min/max lengths and entropy. This tries to use XKCD/936 style passwords where possible, or words and numbers, or letters and numbers or just and letters/numbers/symbols, as needed to meet the specified size and entropy.

We also try to encourage people to use easy to remember passwords that are system generated and random. This avoids many of the common mistakes of passwords, including: re-using the same password; picking easy passwords; and having to write down passwords.

This does mean a few changes. For many passwords you will see the familiar four word passwords that are long, easy to remember and of the order of 48 bits of entropy, but for some things like WiFi passwords you will see a couple of words and several digits to meet space and entropy requirements.

However, as ever, comments welcome...

Distressing

Well, looks like we will actually be going to court over the latest spam.

21 pages, FFS, waffling on about my connection to Dedicated Programmes (from which I buy the email address in question). Whether or not I am an individual subscriber is a matter of fact - it is whether I have a contract with a CP as an individual for the service, simple as that. Even if I was director and shareholder of DP, it would not matter if I, as an individual, had a contract for the service. As it happens, Simon, who runs DP is just a friend and I have never worked for or been a shareholder of DP, but none of that matters. He thinks the invoice was sort of made up - well yes, long before he sent his spam, after previous spams, Simon and I agreed that DP would actually charge me, and properly invoice me, for the email address which he had previously provided free (or in exchange for some help from me). That made it a clear contract, ensuring there was no doubt as to the fact that I contract as an individual for the service and as such am an individual subscriber.

Anyway, I'll have to go through in detail. He has quoted loads from my blog, which is, of course, a breach of my copyright for which I can sue him (sadly I can't in this case).

What is in no doubt is that this is distressing, and it is distress for which I am claiming. I have just checked my blood pressure after looking at this letter from the guy and it is through the roof. I am already on medication for blood pressure. I may take the meter with me when I go to court, as I have to be a bit careful. It will take me a while to calm down now! I can be pretty sure I will lose sleep over this now.

What I have to remember is, that even if the judge throws out my case for some reason, I will have learned something useful and at most paid his costs for getting to court. Small claims has limited costs, and the fact he refused to consider an arbitration call may even get the costs removed anyway, we'll see.

Case on the 15th!

A&A support hours

We have extended our support hours, which are now 8am to 6pm, Mon-Fri, except (English) public holidays. No, not an April fool's joke, honest.

Previously we worked 9am to 5pm, and sales/accounts still do. However the support staff can usually address simple/urgent queries in those areas if necessary.

Occasionally we do have people ask why we only work office hours, and it is worth trying to explain this. Many ISPs do, indeed, have 24 hour telephone support, for example.

For most of our services, there are faults that come in two flavours. Either there is some "shit hitting fan" big issue, in which case we have staff, and myself, getting involved in fixing things whatever time it is, or some individual line fault for DSL. It is pretty rare to have individual faults for VoIP, SIMs, etc, but you can, of course, get line faults for DSL.


When it comes to individual DSL faults, there are a load of things people can do at home/office to eliminate equipment and test for themselves, and we offer various on-line tests via our control pages. This can help resolve things. But the issues that don't just go away, and would require support staff to do something, are almost always something that needs a BT engineer to go out.

With very few exceptions, BT engineers are not going to be going our any quicker if we book them next working day at 9am. So having support staff take calls in the middle of the night would not usually be any help. We also have no intention of farming support out to call centres following scripts.

However, we have decided to extend the hours a bit. The reasons being :-

• BT engineers work 8am to 6pm normally, and so we can help address any issues that come up with an engineer visit, and talk to the engineer or our customer about it at the time. This has already been seen with some 8am visits by engineers who are confused by the notes and need us to explain.
• Starting at 8am gives customers a chance to resolve issues that can be resolved by talking to support before the usual working day for most people. If it is a line fault, that is not much help, but if it is a matter of swap a router or reboot something, we can offer the necessary advice before you have an office full of people that cannot work.
• Starting at 8am and finishing a 6pm allows a lot of people that work during the day to contact us from home where they have an issue with their home broadband. We know some customers appreciate that.
• We have increased the number of support staff, allowing some staggered working hours so that we can offer this.

But please do bear in mind, we do have irc, with a simple web front end, which can offer various help and advice by staff and other customers at all sorts of times. It is informal support from staff outside normal hours, but is usually available. We are thinking of perhaps extending this to be more formal evening irc support at some point maybe, with a rota of some sort.

Obviously we're interested in feedback on how the new support hours work for customers.