These are not the DSL lines you are looking for

Here am I trying to debug WTF I have wrong on my RADIUS code, still, and the log is full of stuff that, well, is a tad unexpected....

I may have to have a chat with BT and/or MDNX in the morning...

2013-04-30T20:40:09.191754Z c.gormless radius-rx 81.145.117.4->90.155.53.21 Platform Accounting Username=MDNX70364@dsl.mdnx.com Calling=BBEU09880595

2013-04-30T20:40:10.271865Z c.gormless radius-rx 81.145.117.4->90.155.53.21 Platform Accounting Username=MDNX135303@dsl.mdnx.com Calling=BBEU09574052

2013-04-30T20:40:10.663198Z c.gormless radius-rx 81.145.117.4->90.155.53.21 Platform Accounting Username=UKDSL00717@vtl.viateldsl.com Calling=BBEU00380785

2013-04-30T20:40:11.570704Z c.gormless radius-rx 81.145.117.4->90.155.53.21 Platform Accounting Username=MDNX135303@dsl.mdnx.com Calling=BBEU09574052

2013-04-30T20:40:12.133318Z c.gormless radius-rx 81.145.117.4->90.155.53.21 Platform Accounting Username=MDNX70364@dsl.mdnx.com Calling=BBEU09880595

2013-04-30T20:40:12.712036Z c.gormless radius-rx 81.145.117.4->90.155.53.21 Platform Accounting Username=mdnx120891@dsl.mdnx.com Calling=BBEU09248797

2013-04-30T20:40:12.893569Z c.gormless radius-rx 81.145.117.4->90.155.53.21 Platform Accounting Username=WOLVS.STORE@LINKTWO.SOLUTIONONE.COM Calling=BBEU04679151

One of those days!

Well, it has been a very very odd day.

I code for a living, but today I have managed around a dozen lines of code, and every single one was wrong, and the fix was wrong and the extra debug I added to find the error crashes stuff. It was a disaster. I had suddenly lost the ability to write printf("hello world\n"); without a compiler error.

I have no idea if this was whacky blood sugar or a cold or what. I think I am feeling better now. I do apologise to Peter who suffered from a sub standard service today - normally someone needing a minor code change because of an unexplained issue with some router sending PPP LCP protocol rejects would be a 5 minute job, new release and sorted. This was like a dozen code releases, each more broken than the last and crashing nicely when you have been running for 10 minutes and think it is safe to put customer traffic on it.

That is so not like me, I felt it worth blogging.

Though, I have to say, someone on irc has blamed the fact that I was, of course, sober (I am during the day!).

Anyway, the good news is that, having found the final (I hope) quirk with RADIUS coding at 5am this morning, and fixing it (about 5 times today), I have a stable release being used by customers and working. Yay!

Now, back to the new logo and updating everything...

Wealthy pensions should hand back freebies!

WTF? Just heard this suggestion on the news!

So people that have worked hardest to earn the most and put that in to the state should hand back some of the freebies they validly get from the state in their old age, like bus passes and free TV licences.

Why is it seen as bad to work hard and make some money? Is that not what the economy needs?

I, and many others, pay a shit load in tax and NI, and the very idea that I should forgo what I am entitled to is offensive. Sorry.

I'll claim every damn penny I am entitled to from the state given the chance, just like everyone else.

We pay for the state, some of us more than others, so why the hell should we not benefit from it!

When I do retire, if ever I do, I may damn well need it. Nobody knows how the future will pan out.

It is not a matter of "the more you pay in the more you get out", obviously. The whole notion of any state welfare is a fundamentally socialist view - but that would mean something like: "Some pay in more than others but we are all entitled to the same out of the state regardless". What seems to be suggested here is one where "The more you pay in the less you are entitled to get out" which is sort of pushing socialism too far and turning it inside out. It sounds like a Monty Python sketch now "stop that, it's silly!".

Spring cleaning my RADIUS code

I explained how code can wear out for all sorts of reasons, and one of the bits of code we us in the FireBrick has finally reached the point where it is better to start again than bolt on extra bits. It is the RADIUS client code.

The original coding used RADIUS, which is a remote authentication for dial-up, as part of the L2TP code (used to handle broadband lines and mobile these days not actually dial-up).

The RADIUS was, itself, a relatively small part of the L2TP code. It is used when someone connects to get a yes/no as to whether they are allowed and a few details like what IP is assigned. It is also used for accounting to report amounts of usage on a connection. The logic is simple, a message each way for each transaction. There is some authentication using pre-shared secrets. It is very much tied in to the sequencing of the L2TP and PPP operations.

So the original code design built RADIUS in to the L2TP code. It was a sane choice at the time.

Over the years the RADIUS code has grown. We have more options on authentication, and a lot of handling for load and scale - like how many concurrent requests can be outstanding to a RADIUS server and how do we pick the server, and blacklist a dead server, and tell it has come back, and so on. Then RADIUS became more complex with support for "Change of Authority" and "Disconnect" messages.

Even so, the RADIUS code was still reasonably sensible, and still only for L2TP.

Then we added VoIP, and want to use that for RADIUS. Suddenly RADIUS is no longer a small side-effect of L2TP, but a more general facility needed by VoIP and L2TP.

That is the time to change the design - now we need a carefully designed RADIUS client module that can provide RADIUS services for any other part of the code in a safe way. Of course, as soon as you think of this you can dream up lots of ways RADIUS could be useful for other things :-)

When you design a module like this you have to carefully consider where you "draw lines". Drawing lines is important! You have to define clearly the areas for which the module is responsible, and what is expected of the application. These have to be logical, clear, and obvious "lines". This makes it easier to test the module, and decide where the code goes, and avoid duplicating code. The idea of creating clean interfaces and partitioned data and responsibilities is actually key to object oriented coding, but is a good programming principle in any system. Previously the RADIUS was all behind the lines of "what the L2TP module does".

Of course, this has meant a couple of annoyances that come out during the code. I have made the module responsible for picking the RADIUS servers, and knowing (from config) the IPs and secrets used for the servers. The application never sees these. The RADIUS module provides functions and macros to manipulate the AVPs (the attribute/value pairs in the RADIUS messages). The module is responsible for retries, timings, stats, blacklisting RADIUS servers, and so on.

One nice thing with an interface like this is some logical state, and responsibility. i.e. when the application asks for a RADIUS request to be sent, there will always be exactly one callback when the response is received, or there is a timeout, or even if the application cancels the request. It is the responsibility of the module to make sure that happens with whatever tasks and timers it needs.

I realised early on that this means some special handling of a few AVPs. For example, the module has to look in to the RADIUS packet and update the "delay" attribute used on RADIUS accounting. This is a number of seconds since the accounting happened, and, of course, if we are retrying after a delay then that needs to move on in time but we can't expect the application to re-make the message with the extra delay.

There were a couple of other gotchas that caught me out - the PAP User-Password field is encoded using the challenge for the request and the shared secret. This is stuff the RADIUS module knows, but the application does not. So the application adds the password "clear" and the module scrambles it. There is a similar effect for received Tunnel-Password attributes that are scrambled in a way only the RADIUS module can decode. This means these AVPs need special handling as they change for each request.

Of course, to make use of the new module means ripping out all of the old code, and that is the scary bit. I have to look at each part, read the comments, read the code, and see if it makes sense. If it does, then I have to work out how it fits in to the new way of working. If not, I have to work out what I was thinking when I first wrote this. That can be fun. Thankfully RADIUS is pretty well covered by some good RFCs. Then I have to test everything, and, of course, make sure documentation is up to date.

The end result looks OK. Lots of testing, but now I have to test on a larger scale and make sure it works properly. Hopefully next week I can roll it out.

But it does mean I can finally progress my VoIP system for use as a large scale voice server with RADIUS based authentication, call routing and accounting.

Ruckus is quite fast...

Apple AP

Ruckus

Neither is anything like direct wired, but WiFi latency is always going to be a factor and I don't really trust speed tests, but as a relative measure that is quite telling. This was to a MacBook Pro.

It is also fun when you have all those SSIDs...

Snooper's Charter

Horrid title, but what we are talking about is the latest in a long line of attempts by the Home Office to have a means to track what people do and where in terms of communications. The Communications Data Bill.

Open Rights Group have more on this.

You can see why they would want it, and the usual example is, of course, a terrorist incident. It would allow them to track who was talking to who, and where they were. Not quite as far as "what they were saying", but quick and easy means to search lots of operators for calls, emails, chats, tweets, and so on.

Of course, fewer people are killed by terrorists than speeding police cars (made up statistic, but I bet it is true, can anyone find stats on this for UK?). It is a scare tactic if anyone talks about terrorist - the clue is in the word (terror). The fact is that terrorists are just like any other criminals, only they are less of a threat that car drivers by several orders of magnitude.

Putting that to one side - what is bad about the Bill itself. It is the latest in a long series of attempts to get this stuff in place. Essentially they used to be able to get phone records, which was fine when all calls went via BT and phone calls were the only real way to communicate in real time at a distance.

The environment has changed - phone calls are on the demise, and handled by many operators and use lots of protocols and equipment. I make phone calls rarely myself, and when I do it is rare for the call to leave a network I or the recipient control, so no logs, even with this new Bill. But I have conversations with friends that involve, in the same conversation, a mix of in-game (WoW) chat where one is playing and one is using iPhone chat app, iMessage, irc, Facebook, phone call, and face to face.

So the idea, from a technical view point, of tracking what people are doing is a nightmare. This is made even more complex with overseas providers like twitter and Facebook.

Of course, collecting data can create a web of connections between people - apparently the president of Nigeria and I are frequently communicating, so that makes perfect sense! Yes, junk mail, will be in their data. Of course it won't (can't) be illegal to deliberately poison their data by creating emails between people, and the Bill is so controversial this could happen using a deliberate virus. Not seeing the content of emails, only the "Communications Data" makes it impossible to tell fake from real communications.

They have suggested that ISPs can intercept the communications to record some of the data. This is not easy, and goes against some basic principles of mere conduit. It is also impossible when there is encryption, which, since this whole idea kicked off has moved from rare to standard to mandatory for many applications and web sites.

This will be the first time the government is forcing private companies to snoop on citizens who are not even suspected of a crime, store that data, and allow the government to search it at will. It is a proper "police state". I was never too hot on history at school, but even I know where this ends up.

Lets not forget that all this snooping invades privacy of innocent people whilst doing nothing to catch "bad guys" who have the slightest clue. My understanding of things around 11/9 are that terrorists knew tricks then, like making draft emails and not sending them but logging in to read the draft and deleting it. I am sure that these days they will easily be able to use a range of encrypted communications means that will be off the radar. It is so much easier now.

They need to understand that they have lots to technology - Yes, they want to be able to snoop, but they can't. It won't work. This is not just something like the German's using Enigma in the war - the basic technology for powerful encryption is in every phone in everyone's hand right now. Make the citizens the enemy here and they have the means to fight back. Make something illegal and it goes underground (the US learned that with prohibition). They have to accept that people can, and will, communicate.

What is slightly ironic is that people also state so much publicly these days that the stupid criminals (the only ones they have a hope of catching with this Bill) will probably boast of their crime on twitter of Facebook for all to see.

• This Bill is pointless as it cannot actually stop people avoiding it if they want
• This Bill is costly for the tax payer and the estimates seem way off already
• This Bill is morally wrong as it is routinely snooping on non criminals
• This Bill is a huge invasion of privacy that will be misused in the future

We need to make the politicians realise they are wrong. They work for us, not against us. We are not criminals or suspects and have a right to privacy.

If this goes ahead we'll have to get a lawyer on staff to help us structure new services that bypass these new laws. We are putting encryption in our FireBricks. We can set up off shore Internet access endpoints via arms length companies. It can be done. We can make a way to provide proper Internet access that is snooper free. Commercially we should welcome this Bill as it allows us to sell such services, but we would rather not have to.

Making a ruckus

Well, using one anyway - we have finally switched to a Ruckus AP in the office.

We have gone for their entry level, two band, ZoneFlex 7363 model from LinITX, who were very quick and efficient, and it was on my desk 9am next day.

This is a module that can work stand alone (i.e. without a separate Zone Controller) and is more than enough to cover an office of our size.

First impressions are not good but this is primarily because of the web interface. It is unbelievably slow, and has a horrid way of working - every form appears blank, and several seconds later (even if you have filled in several of the fields) changes to have the initial values loaded (zapping what you have typed). It makes any configuration really painful. It also takes an age to boot.

In some ways I am slightly spoilt by using a FireBrick - boots in a fraction of a second and web interface is nice and snappy. But still, this ruckus web interface is hard to use and frustrating.

It is a let down, because technically it is looking good - 2.4GHz and 5GHz each with 8 separate SSIDs which can be linked to VLANs and a load of features we're not using (RADIUS authentication, L2TP tunnels, etc). The coverage for the office is good even from the one AP. The main advantage is associations and coverage - the apple APs were not quite as good covering the office, and only did 50 associations - some APs do as few as 12. With many staff having 3 or more WiFi devices on their person, and training courses, and so on, a good AP is worth while.

But the whole idea of an entry level device is something that looks really impressive and introduces you to the product range - makes you want to expand to the bigger and more expensive boxes. At that, the web interface is a total fail.

My colleagues are telling that nobody uses the web interface - they use a zone controller. But this is meant to be usable stand-alone - that is the idea of an entry level device. It may well be that nobody in Ruckus ever uses them stand alone and that is why they don't realise how bad it really is.

Anyway, lets see how it goes - we have turned off the apple APs (which are good) and we are using the Ruckus. We had 4 apple APs, so this is not that much more expensive as solution. Time will tell how well it really works with a range of devices.

I suspect this will make a good product to recommend with our Office::1 package.

I now have to resist the temptation to do loads of joke SSIDs on the spare config slots. It has some annoying validation on SSID which won't let me do a Dr.Who SSID, and I have yet to work out how to bypass that.

Oh, and of course, it goes without saying, it does IPv6 :-)

P.S. the CLI (via ssh) is quick...

FTTC: Wires only or not wires only?

FTTC is currently an engineer install with a BT provided modem.

As a "service" from BT this works as an Ethernet service - BT providing the ability to send/receive Ethernet packets from the port on the modem through to the kit in the exchange (and then on to us via PPP and L2TP).

They are, however, trialling a number of options on this. Engineer install is expensive, so how can it be done without an engineer visit to your house?

There are several options, but the main two being considered are "wires only" where we (or the customer) provide the VDSL modem, or a non-engineer install where the customer is sent a BT modem to plug in.

Some people have asked us what we are doing on this, and at the moment the plan is to try the one where it is still a BT modem. We don't know the pricing yet, but there is a reason for this.

With ADSL, which is wires only, there is a whole system of Special Fault Investigation engineers that has been set up and changed over the years - this is entirely as a result of the battle between ISPs and BT over the "grey area" of the DSL circuit. If the DSL is not working correctly it could always be the modem at the BT end, the wiring in the BT network, the customer wiring, or the customer modem. There is no way to tell. So this leads to BT automatically blaming customer wiring and equipment and charging for an engineer visit, even if we have provided replacement known-good kit, and so on. The whole nightmare of SFI continues to this day with every ISP having staff or even a whole department, to dispute BT bills for SFI changes that are not valid.

With FTTC at present, this does not happen. BT are responsible for everything up to the Ethernet port. They can see if there is a problem, test for packet loss, test all of the VDSL parameters, and they have to fix if any of that is broken. There is no grey area and faults almost never fall in the cracks between BT or customer responsibility. This is good news for everyone, us, BT, and our customers.

So, we are very keen to maintain a service where BT are responsible for the modem as well and cannot ever blame customer equipment for a fault. Actually, we'd love an ADSL service with PPPoE bridge/modem that is BTs responsibility, ideally one that is built in to the master socket, but we'd certainly like to retain this for FTTC.

Ultimately, even now, customers could use their own VDSL modem, as long as they are prepared to put the BT one back in when investigating a fault.

Does that make sense?

Special nutters even in 1st class

Walk in to 1st class on the train, sit down at table, get laptop out.

Someone else in carriage, well, only person, right at end of train. She gets up and asks "are you going to be using microwave technology?". Err, what? "Well like mobile phones or wifi - I am electro sensitive".

So I explain that there are mobile base stations all along the route and they will constantly pumping out a lot more RF than I could, so you will have a problem then!

She gets some sort of meter out and starts on about 15V/m or some such and insists she knows what she is talking about. This is while I am not in a call and not transferring any data, I might add.

I open laptop, she leaves, clearly unhappy. I am, however, a tad speechless. I was not prepared for that. Had I known a nutter was on the train with an RF meter I'd have brought my spectrum analyser with me.

I bet she was only in 1st class to be at the far end of the train, i.e. right next to the driver with his mobile phone, but she can't see him so that probably does not count.

I am surprised they let them out in public!

Update: I have to add some extra strangeness to this post, having never met or heard of such nutters before, turns out that Alex was given a card last night (left under his wiper).

What is the betting they sell expensive RF meters.

P.S. Obviously the irony of "turn off all those things that let you get to web sites and email and contact us, err, via, err, web site or email" is not lost on me, nor the stupidity of a proper(ish) domain and then an email that is not under that domain but tied to an ISP. I also like the idea of turning off phone masts! They clearly lack clue.

Update: The train filled up rather at Didcot Parkway, lots of people with phones, so she stormed back in to 1st class, sat as far from me as possible, and proceeded to cover herself completely in a white cloth. She is clearly even more special than I thought. I hope my camera did not steal her soul.

"if you paid more this would not happen"

One of the problems we face on a day to day basis is customers who's broadband service has a fault.

An individual broadband line can work for many years with no fault, but with thousands of customers our support department deal with faults every day. The most common types of faults are physical faults on lines or equipment. This is no different to telephone lines that can fail due to wear, corrosion, even copper theft. In some ways broadband is more susceptible to faults because it uses more of the phone line in a way - i.e. you can have working phone but non working broadband.

One of the challenges the support staff have is handling a customer for whom their Internet connection is critical for their business. It is increasingly common that a business will grind to a halt with no Internet access.

It is a fact of life that putting copper wires in the ground, you will get faults. Some of the wiring is decades old. It has many joints that are in various types of cabinet and distribution point. Things can flood and break. Thankfully some newer fibre technologies increase reliability, but for normal broadband this is the state of things. We try very carefully to explain this when people take a service - that it can fail and can take days to fix in some cases.

So when someone has Internet that is critical for business, it makes sense to ensure you are not relying on a single broadband line which can fail. You need some sort of backup. That could be as simple as an iPad with a data SIM or a mobile you can WiFi tether - if you are a small business. For larger businesses you need to consider multiple lines, 3G backup and so on. As an ISP we offer a range of higher availability services from multiple lines, our Office::1 service with 3G backup, and even direct fibre connectivity. There is a choice of price and level of service and reliability. A business can make a simple commercial decision as to what level they need and can afford.

But at the end of the day, if you pay more for a higher availability service, the chances of a fault stopping your Internet completely is greatly reduced.

I say reduced, we had a case last week where a copper theft took out hundreds of lines, including both lines for a customer that had two lines. 3G backup is the next step, but there are, of course, even less likely scenarios where all links can fail, and even simple things like power cuts can happen. It is all about managing risk.

What is strange is that, when you explain this simple fact of life to a customer who only has one line for their business: that "if you paid more this would not happen" (as a customer put it), they get all indignant and cross. What is that about?

I think the best we can do is try and ensure staff are as tactful as possible.

But if you have a business that relies on Internet, please do think about what backup you have for that.

Viewing a web page is not breach of copyright

Finally, some sense...

"...it has never been an infringement, in either English or EU law, for a person merely to view or read an infringing article in physical form."

Basically, that "viewing" something is not a breach of copyright - you can read a book without being in breach of copyright.

It was considered that "browsing a web page" made temporary copies that were considered a breach for various reasons. There are clauses in the legislation to allow these transient copies (on screen, cache in memory and disk, etc) but only for lawful use, and the argument was that reading the web page could be unlawful if not licensed. But the judgement here says that "reading" the information in a web page does not need a licence, and so is lawful and therefore so are the transient copies made as part of that process.

Choosing to download or print a web page is not the same, that is protected, but simply viewing a web page is not a breach of copyright.

Yay!

http://www.supremecourt.gov.uk/decided-cases/docs/UKSC_2011_0202_Judgment.pdf

Also http://www.out-law.com/en/articles/2013/april/unauthorised-browsing-of-copyrighted-material-online-is-legitimate-says-uk-supreme-court/

"Real" Grandad

Well, I was not sure whether to say this or not, and I can't post any of the cool pictures yet (at the request of the parents), but I am now a real Grandad as of yesterday afternoon. Lewis is doing well, though needed antibiotics and is being guarded by Ignis looking over his cot.

This does make me very proud of Jacqueline. One of the few things we can do in life that have an impact in to the future is to have children and grandchildren and great-grandchildren and continue our "line"...

I say "real" because for some years now I have been a "step Grandad". I don't mean to have a go in any way, or suggest that my step grandson is somehow inferior. It is just a fact that he is not "mine". It is not a fact I can change, or really just ignore. Whatever I do I know that fact. It is, sadly, one of those complex areas of family relationships which are made even more complex by second marriages and step children, half sisters and brothers, half aunts, etc. The problem is that there are, it seems, some level of rules on what one can and cannot say without upsetting one party or another. To be clear, I am not trying to upset anyone. I am, if anything, trying to be "technically correct" (the best kind of correct), and not have a go. Really. There is a small boy that calls me "Grandad" already, and he is cute, and very much my grandson. It is not his fault that he is my wife's grandson and not actually mine, and not a point I would ever try and confuse him with. As far as he is concerned I am "Grandad" and I am happy to be.

So, well done Jacqueline. I say "well done" as it was not easy for her. A week in hospital with typical vagueness of doctors and consultants scaring you and then doing nothing. Comments that he was too big, or lost weight, and all things that make a mum scared shitless. It is not an easy time. She was in labour for over a day, and exhausted, but did not complain. She was brave and strong. It was very nearly a C-section, but not, thankfully. Sandra had her chance to speak her mind to some doctors in Frimley Park - she was not impressed with how they were handling things at all. I did what I could do, which was to keep out of the way and manage to feed myself staying at home. Needless to say I did not get much sleep either.

At the end of the day, it does make me feel a tad older. My parents are finally great-grandparents, which makes them proud, and probably feeling a tad older too.

When I am allowed, I'll post some pictures.

Yes, & is a BACS character

BACS allows some specific characters in the coding of BACS transactions, used for payer name, beneficiary reference, etc. It is A-Z, 0-9, . (dot), - (hyphen), / (slash), space, and & (ampersand).

The catch is that & is special in HTML. So needs escaping, obviously.

So I sent a 1p payment to a customer for fun. I sent from AA&REG

This is what he saw:-

Naughty Bank of Scotland.

Oddly, on the CSV download this is "AA AND REG" which is even stranger.

FYI Barclays seemed to escape stuff properly.

Banks still have a lot to learn!

Tackling junk calls (ePetition, please sign)

One of the issues with junk calls is that the enforcement is by ICO who seem rather ineffective at discouraging junk calls.

There is one other measure in the legislation - people who are the victim of junk calls (when registered in the TPS) can sue for damages in the county court. But what is the damage for one junk call or even a couple from the same caller. Bugger all. So it is (a) nor worth sending a bill or suing, and (b) likely to be hard to justify any damages you do sue for if it goes to court.

What I think would be effective is if thousands of victims of junk callers were able to do something worthwhile against the junk callers - death by a thousand cuts.

So I have made a proposal and put it in to an ePetition - please sign if you agree.

Amend The Privacy and Electronic Communications (EC Directive) Regulations 2003 adding a new clause:-

"30(4) Where such procedings make a claim for damages not exceeding £50 the claimant will not be required to justify the value of the claim, only that they have suffered some damages, inclding mere inconvenience, as a result of a breach."

Doing this will allow anyone suffering junk calls to invoice the caller, and then pursue a county court claim, for £50. The callers will have to settle without court as they know they would lose, so no burden on the courts.

I feel that thousands of victims of junk calls invoicing £50 would be very effective, as well as satisfying for the victim.

Update: Don't forget that this would have the same effect for junk emails as well.

What the hash?

OK, this may take a little explaining, but I think that all the huge hassle we have had to sort new BACS systems may have been totally and utterly pointless crap.

How it works
We do have to send HMRC details of payroll each month (well, on or before each payment made under PAYE). That is the new system. Previously we did this once at the end of the year.

It uses XML and stuff to send the data, and on its own this is not too bad. We have to do all the work for PAYE and make pay slips and so on - the extra work to effectively make an XML payslip and send that to HMRC is not big. Indeed, I got most of it sorted in a few hours at the start of the week. As ever I have spent a lot of time polishing up the old code for PAYE (see other blog post of code wearing out).

But yes, on its own, simply sending an XML copy of payslips as we make them to HMRC is not a huge task.

What is hard work is that every BACS submission software package and BACS bureau has also had to change to allow "BACS field 7" to be populated. A previously unused 4 character sub-reference. This even led to Lloydslink shutting down their BACS bureau. It has taken a lot of time and money for us to do this all.

The reason is that, when we send the data to HMRC, we can prove we paid the employee. This is done using a hash.

The way it works is that a one-way hash is made of some data in the BACS payment we make. BACS do this, when they get the payments, and they send the hashes to HMRC. The theory is that HMRC do not get the whole of the bank details (privacy issues?) just the hash. We do the same calculation and send the same hash in the submission to HMRC. HMRC can match them up, proving we paid the employee.

The hash is made from fields in the BACS payment. These include our sort code, the recipient sort code, and the amount we are paying. They also include this sub-reference, which has to have a "/" at the start (so BACS know it is for HMRC to get the hash) and there random characters. The random characters mean the same person paid the same amount each month does not get the same hash. They could, of course, have simply included the date to avoid that, but maybe that was too risky as BACS could be run early or late for lots of reasons.

Clever stuff, you may think. Including the amount in the hash means HMRC know we paid the right amount to someone for payroll, doesn't it? The theory is that if they cannot match the hashes then they consider that more of a risk in their risk management.

The problem
There is, of course, a huge hole in this. All the hash actually proves is that we made a payment. It is not likely we could make a matching hash for some payment we did not make. It could be any payment to anybody and any amount, as long as we made it, and and therefore we can work out the hash for the payment. There is no way for HMRC to check the hash we send them is right as we don't tell them sort codes, or the sub-reference. All they can check is that it matches some payment that went through BACS.

Wouldn't BACS tell them the amount? Well maybe, but if that is the case why include the amount in the hash?

Wouldn't BACS tell them all the details, sort code, account, amount, etc? Well maybe, but in that case why have a hash at all - why not make the submission to HMRC include the sort code and account of the employee? That would have meant we would not have had to change BACS at all.

In fact, HMRC state that they will not have any access to the source information from BACS (such as bank account details). But to check I have FoI'd them. I'll FoI how much all of this has cost later!

Basically, when making the submission to HMRC, as long as we have a means to make a payment of some sort via BACS, even if it is 1p (or indeed, the correct amount) that is paid back in to the same account it left, and we can work out a hash and include that in the submission, it will match up on HMRC's side. The actual payments to staff could be completely separate, different amounts and times and anything else. The very issue one assumes they are trying to address by matching up payments.

Basically: It is trivial to fake the BACS payments with matching hash, independent of actual payroll payments.

So why the hell did they bother?
That is the real question. Why make a half hearted system that has caused a lot of people a lot of work for no real gain?

Either don't try and check payments, or check them properly by requiring the actual payment details in the submission (sort code and account). This faffing with hashes is a total waste of time.

PAYE coding

I have to say, as I work through this nightmare of PAYE coding crap for HMRC's new RTI stuff, I am consoled by the daily Dilbert. It shows me what I am up against.

SIP progress

Well, I am finally coding again, and have been bogged down with BACS stuff and HMRC stuff, and even some FireBrick stuff, and I really have to make some progress on the FireBrick SIP code now.

We have been using in the office since last year some time, and it works really well. The features so far make up a good, scalable, PABX system. We can expand on those features over time, but we already have hunt groups and time profiles and fallback groups and busy lamp fields and call steal and transfer and so on. A good set of PABX features.

One small point has been integration testing, and I am pleased to say that it is going well. We have customers starting to use the FireBrick as a PABX now, and getting useful feedback. Even the entry level FB2500 has the PABX features.

We know it works well with a range of handsets including Snom (on which busy lamp field works too), grandstream, gigaset, and I think a few others. Obviously if anyone has anything they can test, let us know. We have tested on a few IPv6 handsets now too and that works well and maps the media to/from IPv4 as needed.

The server side has not had a lot of testing until now as people have been using with A&A VoIP, and we know that works. Turns out we were missing some of the Record-Route logic for use with proxies. I was hoping to avoid a lot of that by being an endpoint only and not a proxy. We had included the main areas of this but not all. One of SIPs most annoying features is the route set logic which has to be stored on the dialogue. Well, we have added that now, with some careful reading of the RFC and testing.

The result is we have it working with sipgate and gradwell nicely. What is fun is that we even have it working on BT with credentials that normally go on a home hub. Sadly this is a slightly bodged config with a faked DNS reply added as BT talk to a proxy but use a different hostname in the actual messages, one that has no A or SRV records. They could have done this right so simply, and just need a DNS entry. I may try and make the FireBrick config cater for this though, as it seems that some other devices will do the same (talk to named proxy but use different name within the payload). I am sure we can do the same if that is what devices usually do.

The plan is to test more handsets and SIP services - apply any fixes if needed - and write up configuration examples on the web site. We'll also look at any changes to make things more intuitive in the config (always a challenge for SIP).

I then need to work on the second phase of development, starting with a re-write of the RADIUS client code so that it can be used for SIP as well as L2TP, then doing RADIUS based user registration and call routing and CDRs... We then have various fun features, like call recording, voicemail, and so on, to bolt on. Maybe even conferencing if I feel adventurous.

Watch this space, as they say.

HMRC and summer time

So trying to access HMRC DPS service - this is basically a system to get the notifications (e.g. tax codings, and so on) from them.

Looks OK, SOAP based, XML (D'uh) so simple enough.

But for authentication they need you to get a "token", which is a chunk of raw XML sent in a CDATA response. Eeeeewwww

Well, I tried that, and parsed it, and included it in the request (with namespaces and all) and it failed. Given the way XML canonicalization works I expected to have to send character for character "as is". That is a horrid bodge to my XML library, but what the hell?

But still not working.

So why?

Well, not sure yet, but maybe HMRC do not understand summer time.

A request for a token at 17:05 BST gets a reply that includes...

IssueInstant="2013-04-06T16:05:00.185Z"

Well, given that I just pressed the button when my watch said 17:05, that is pretty good, only 185 ms out :-)

For those that do not know, the Z on the end means Zulu time, i.e. GMT or UTC, not BST, so 17:05 being 16:05:00.185 is good.

But hang on ...

AuthenticationInstant="2013-04-06T17:05:00.185Z"

Thats an hour out... Worrying...

Then this bit...

<conditions notbefore="2013-04-06T17:05:00.185Z" notonorafter="2013-04-06T21:05:00.185Z">

That is not good. It looks like we can't use the token for an hour.

How hard is it to understand the basics of British Summer Time, FFS.

Update: Even waiting an hour did not help, and my error was using wrong servers and SOAP action in wrong places. They did not document simply what they are but provided an XML definition. They then use test system URIs in the namespaces on the live system which fooled me, but turns out to be what they want for it to work. Anyway, we now have a working DPS system, and all ready to do an Employer Alignment Summary this week ready for our first RTI submissions this month. Overall it has been probably less than a day's coding, and getting replies from HMRC test team on Sunday morning is impressive. I am sure there are several more hurdles though...

Nationwide and DDs bouncing

So, anyone else with a Nationwide account had this.

You pay someone by DD, and some of the DD collections are bounced by the bank with "AMOUNT NOT DUE".

The DD rules are clear - that error is for when the collection is within 2 days of the DD being set up by AUDDIS (the paperless system for DD instructions) or because the payer (you) say you did not get the right notice (i.e. collected too soon).

As a DD collector we have had this a few times recently with customers with Nationwide accounts. We are trying to reverse engineer there rules. We tested with a member of staff after a couple of customers had issues.

It looks like they bounce any DD that is 2 or 3 working days after the first collection. Well, that is the guess so far. It could be some "rate limiting" thing, e.g. more than 3 DDs in 3 days or some such. We have more tests to do.

The problem is that we launched a new tariff with very strict credit control - bounce a DD and you are cut off until sorted. We have had to somewhat relax that logic as a customer with a Nationwide account apparently bounced a DD when in fact it was all done by their bank.

Having some sort of logic for rate limiting is not totally daft, although the DD guarantee makes it a tad redundant - but we have people who sign up, are invoiced a connection, and a few days later invoiced an ongoing bill, so two DDs close together. In some cases they buy another service like VoIP, and bingo - three DDs in a short space of time. All notified as per the DD rules.

The real problem we have is the bullshit from Nationwide. Sorry, but that is what it seems like to me. It is bullshit - and does not stand up to simplest of checks.

1. They tell customer that A&A did not send the collection, our fault. We have proof they rejected the DD, so that is a lie.
2. When confronted with the XML of the bounce from BACS they say we only "set up" the DD on the first payment, and not weeks before. That is also a lie.
3. When confronted with a screen shot of their own on-line banking showing that is a lie, and DD was set up before, they invent new problems. They invent the idea of a DD instruction having an "end date".

They are totally unable to explain their actions. The last straw was this imaginary final collection date ("end date"). We have worked it out, it is on their screen as "last collection", we suspect. We even asked for the date for a load of other DDs set up on the account, and all were in the past. Clearly "last" collection is not a "final" date but just the date the last collection was made. But it shows they will invent bullshit to excuse themselves. They also blamed "keying errors" by A&A or our sponsoring bank, but could not explain the two payments that did go through. They seem adamant that the sponsoring bank are involved in AUDDIS submissions even though we are "direct submitters" to BACS/VOCA.

So how to fix - we are working through channels, via BACS (who confirm that error is only valid where payer rejects or within 2 days of AUDDIS), and via sponsoring bank. No idea if it will work.

However, we think we may have a simpler approach. It is a bank error, so covered by DD guarantee for consequential losses. So I am tempted to say that the admin fee for a bounced DD with that error is £1000+VAT. That way, anyone suffering this gets to claim from their bank, involving the financial ombudsman if needed. If we get any that are actually paid, we'll split it with you as a £500 credit off future service bills. Anyone wanting to try this is welcome to talk to me on irc - we'll formally agree the charge and then submit the fully notified DD collections.

Wunch of....

P.S. it is probably not valid to make up an admin fee to share with our customer, sadly, but I think we should at least charge our standard admin fee if they don't sort it.

HMRC RTI Hash up

So, just started looking at the HMRC RTI stuff. I mean, how hard can it be...

The first step is that to send the data to HMRC you have to go via the government gateway. The whole lot is XML, which is, itself, not an issue. XML is fine, it is the way people use XML that drives one mad.

So, gov gateway uses a SOAP style envelope posted via https - simple stuff. The envelope includes login credentials which can be plain text, MD5 or proper certificate signed. MD5 is not much better than plain text as it can be replayed, but does not expose the original password. In any case it is all over https, so not too bad either way.

The MD5 is interesting as the steps are (a) convert to lower case and UTF-8, (b) Make MD5, (c) encode as base64. So I wonder, why make the password case insensitive? Who does that? Then I pondered more - UTF-8 means they understand it could be non ASCII. Do I have to convert an upper case Omega to lower case? What of accented latin characters, or latin characters with accent modifier characters. Doing all that would be a lot of work. I decided not to bother asking and just use tolower() and hope for the best :-)

Then we get to the actual payload, a single XML object within the in the envelope. Again, standard stuff, but for HMRC this needs a hash. This time it is an SHA1 and base64 coded. This would not be too bad if it was an SHA1 of the data as sent, but there are some gotchas.

For a start, the hash is within the message, so you have to make the message, then make a hash, then remake the message with the hash within it. You have to be sure of making the object the same both times (which is fine unless your code uses counts of namespaces usage to control local tagging, etc, and adding one more item could change things).

What is worse though is the output has to be canonical XML. This would be fine if it was just textual normalisation, but it is not - you have to order the attributes in the elements alphabetically. Well, again, not too bad until you realise the order has to use the original URI from the namespace prefixes to order. This means full XML parsing in order to normalise the output. This is fine except you have to retain all the extraneous spacing which my XML parser does not do, as spacing between elements is used in the hash. Anyway, as we are only generating this we can generate normalised canonical XML in the first place and so make a hash.

So far so good. You make a hash of the HMRC message, embed in the message, and then pass to the gov gateway handling code to include in an envelope and send. Well, no! The hash is not over the HMRC message (i.e. from start of <IRenvelope ...> to end of </IRenvelope>) it is over the encompassing element in the sending envelope. This means the embedded IRmark hash has to be worked out in light of the sending protocol. It is just adding <Body> to start and </Body> to end, in theory, except it has to know the spacing that the wrapper will use between <Body> and <IRenvelope>. Also, just for fun, the <Body> is not as sent, but with the xmlns needed from its parent envelope. In short you can only add the IRmark in the envelope handling code so mixing the logic between layers - messy.

Anyway, once you get passed all that, there is another hash. This time for the BACS payment, but not MD5, or SHA1, no, this time SHA256. And not base64 coded, but lower case hex coded. Just to add more to the mix.

Anyway, now to integrate with payroll...

Update: This is all simple stuff, and HMRC were keen for us to be ready by 6th April, but they are closed! Their systems are down for maintenance! Arrrg.

April fool date command

Why does the linux date command not understand tomorrow or yesterday properly.

A script run at 23:00 on 30th March checked date -dtomorrow +%k was 01 to confirm it is running on last day of the month, but the damn date command just adds 24 hours, instead of getting to same time the next day, resulting in just after 00:00 on 1st April. The same script then ran 23:00 on 31st and ran again...

A script ran around 00:30 this morning that creates a directory based on date -dyesterday again got the wrong day...

Both scripts being fixed. But really... Arrrrrrg!