Efficient pdf and postscript manipulation

We send invoices as a PDF, and we sometimes attach a newsletter or similar to the invoice. The catch is that it can make the PDF rather big.

So I have been experimenting. I have a relatively simple SVG file created in inkscape, and including some SVG from the Internet Society (it is about World IPv6 launch).

The SVG is 65k
Saved as a PDF it is 67k
Saved as a PS it is 1.4M
Saved as an EPS it is 1.3M

The problem is that the postscript is not quite right for some font related reason when move to another machine and run through ps2pdf. So needs "normalising" which means using ps2ps. Using ps2ps is usually the best way to get "clean" postscript that works on anything.

ps2ps from the PDF is 2M
pdf2ps from the PDF is the same, 2M, as you would expect
ps2ps from the EPS is 2M
ps2ps from the PS is 2M

Basically, all of these use ghostscript to process the input and make the clean postscript output.

Doing the same on another machine makes 31M files not 2M. Hmmm. The first machine has ghostscript version 9.05 and the second has ghostscript version 8.70. It seems the newer one is better - good.

But there is another app called pdftops, and using that on the PDF makes a 3.2M PS file. But doing the same on another machine makes 168k PS file. The large file is from an older pdftops (0.18.4). The newer version (3.00) makes the small file. Again, the newer version seems better.

Oddly the resultant PS from pdftops seemed to be wrong as well when converted to a pdf. So ps2ps to the rescue again.

Using ps2ps on the 3.2M file created a 692k file
Using ps2ps on the 168k file created a 93k file

Sounds good.

Finally, we want a pdf as the actual output, and ps2pdf on the 93k file created a 54k PDF.

It looks OK, but some subtle shading or transparency was lost in the process. That I can live with! It is not rastered by mistake or anything nasty like that.

So, the whole process to a 65k SVG and made a usable 54k PDF.

Even so, I am not convinced the final PDF has the text as proper font characters rather than outlines, so I am going to do a bit more experimentation. It would be really nice to get them as proper font characters so they are selectable and searchable and even smaller in size...

Update: The reason the output of pdftops did not seem to work was down to some other postscript I was using with it, I think. Seems odd. But on its own, it works. So the trick is export as pdf from inkscape, use the new version of pdftops to make postscript (not pdf2ps). Play with postscript as needed to make final composite document, and they either print or ps2pdf to go back to pdf. The final result was 44K PDF with searchable and selectable text. A tad better than 31M.

Technically correct (union flag)

One of the small jobs I had yesterday was the new router cards.

We put these in a small clear plastic wallet attached to the routers we ship.

They contain the login details for the line, the router and wifi. We have gone for a single sided design this time.

This should have been a simple job!

First off, I thought why not include a 2D bar code that has the WiFi connection details. It seems many phones will now read these and pick up the SSID and password from them. I already have an IEC16022 (Datamatrix) encoding app I wrote many years ago. Simples!

Except it turns out the code has a bug. Even though published long ago and used by many people, it does not code EDIFACT correctly. In the case of the upper case characters and punctuation used in the WiFi password data, EDIFACT was the best coding, and it just made gibberish. So that wasted an hour or so fixing it.

The rest was easy, but when I had finished some bright spark (Steve, I think) suggested that we could put a union flag on the card as it is the jubilee. Fair enough. How hard can it be?

Uncharacteristically, as it was a gimmick and cosmetic, I just made something that looked like the union flag. Of course, and quite rightly, I was immediately slated for this by the denizens of irc. Sorry everyone... What was I thinking?

So I went off and found the 2:1 ratio union flag design, and put that on. It kept them happy. However, I thought I should dig a bit deeper. Thanks to this site.

Interestingly the union flag design is specified in such a way as to allow any ratio of flag, even square. I never knew this, but it is not just one design that you stretch to fit. It is different when drawn in different ratios. In the case of this card the ratio is not 2:1 (or 3:2 or 5:3 or one of the common variants) but was around 200:127 as that is the space I had. Actually the ratio is dependent on the left margin I set, and can be changed easily in the design now.

All of the sizes are based on the height (or as I believe it is called, the width where what we would call width is the length). The top to bottom distance, OK!

Postscript is really good at this stuff, and makes it very easy.

First, we define w and h as width and height, and make some derived values. d is diagonal distance; a is angle of diagonal; u is a unit which is 1/30th of the height h.

/d w dup mul h dup mul add sqrt def
/a h w atan def
/u h 30 div def

We make the rectangular area as a clip area for the drawing:

0 0 moveto w 0 rlineto 0 h rlineto w neg 0 rlineto clip

Then we may as well fill the whole flag blue:

0 0 102 256 div setrgbcolor % blue

fill

Then the white diagonal cross which is 6 units thick and from corner to corner:

1 setgray % white

0 0 moveto w h lineto 0 h moveto w 0 lineto
u 6 mul setlinewidth stroke

Then the thin red diagonals which are 2 units thick. We need to rotate to do this right:

204 256 div 0 0 setrgbcolor % red
u 2 mul setlinewidth
gsave a rotate 0 u neg moveto d 2 div 0 rlineto 0 u 2 mul rmoveto d 2 div 0 rlineto stroke grestore
gsave 0 h translate a neg rotate 0 u neg moveto d 2 div 0 rlineto 0 u 2 mul rmoveto d 2 div 0 rlineto stroke grestore

Then the England cross white border:

w 2 div 0 moveto 0 h rlineto 0 h 2 div moveto w 0 rlineto
gsave 1 setgray u 10 mul setlinewidth stroke grestore

And finally the red cross (we gsaved it above):
 
u 6 mul setlinewidth stroke

Job done. Flag poll assumed on left.

Oh, and add some text to explain why on earth we have but a flag on there in the first place :-)

Note: You probably want to wrap this all in a function with gsave/grestore, etc.

And in different ratios, it looks like:

Well done Andrew, Jimi and Paul

Yesterday was rather busy - starting with a power outage at the data centre the night before.

Whilst the outage was under 4 minutes, and equipment should all start up and carry on after an outage, it seems four servers did not. This makes me wonder if it was a power spike, but I have not seen the report from Pulsant yet.

Most things did start working, as you would expect, and our routers recover within a couple of seconds of power being restored. One server that failed was a spam filter, so just meant we had less capacity for spam screening. Another was one of the outgoing mail server pool, and again we can work with one being off line. Finally one of the disk servers (with all my stuff on it, as well as customer voicemail recordings) was down. One server handled usage recording for calls and broadband, so people got some for free!

Jimi and Paul worked on these issues over night, and Paul and Andrew were working on them during the day. The mail server queue was cleared. The disk server fixed. I was able to ensure the billing system was working correctly once the usage recording database was back. It was a busy day all around.

The day did not end there as something blipped last night in the other data centre with two routers having trouble with a switch. Thanks to Paul and Jimi for work on that and fixing it within a few minutes. Again, most systems re-routed automatically, as they should.

Well done everyone.

Diablo ++

OK, James copied some cache directory and as if by magic it works.

Well, works as far as saying the EU servers are busy... FFS!

With 6 digit error code, why the fuck can they not tell "you got your username and password wrong" from "we could not access some local cache file"...

How mental.

Anyway, if ever it works, I may play. I already had to pay for James to get a copy :-(

That's how to fix a bug?

A friend of mine ran in to a snag with some code using mysql client libraries. Asked me for help. valgrind explains it is a memory access deep under mysql_real_connect. The odd bit is that the code worked on older machines, and even moving the binary from an older machine to a new one worked. Also, other programs worked even though using the same library and same function.

So, I explained SEP fields, and LMGTFY, and he went off to investigate. I am impressed, he found what was different. The compiler arguments!

The difference was that mysql_config appears to include -rdynamic. Without this, the code works. That alone is a tad odd, and worthy of more investigation.

However he also found a bug report http://bugs.mysql.com/bug.php?id=39175

And that got me puzzled. Do have a look at it.

The bug report is 2008, but this seems to be a recent problem. However, oracle (or was it not them back then?) confirmed the report within 40 minutes. yay!

Over a year later (yes, a year) they say that to use his contribution they need the original reporter to sign a contribution form. A month after that they suspend the bug report.

I thought "What contribution?", and then I realised the report does indeed include a "suggested fix" changing @LDFLAGS@ to @SAVE_LDFLAGS@ to reverse some previous change.

It is clear from the report (saying that between one version and another they changed from @SAVE_LDFLAGS@ to @LDFLAGS@, and that was wrong) what the fix should be, but I have do wonder... If the original reporter had left out the helpful "suggested fix" part, and just made the report, then would that not have counted as a "contribution" and meant the bug was actually fixed?

It sounds like sending a "suggested fix" is actually less helpful than not!

Way to go oracle - classy bug report handling.

Now to work out how we work around this cleanly and why -rdynamic is breaking things!

Diablo -1

OK, having tried everything, and no reply from Blizzard on a ticket, we went for a complete re-install of windows from scratch and reinstall of Diablo 3 from scratch.

We even skipped the nice 3D graphics install and everything else - just totally bog standard windows install and then Diablo 3.

And no, it is being an arse. Error 315300 wrong login.

To be clear, we have tried with and without an authenticator set. If I put the wrong username or password it says no right away, but with them right it goes on to ask for the authenticator. That means it is checking they are right for both username and password. If I don't have an authenticator it does not ask or work.

We tried from another PC, and that works? Well, actually, if we try from my machine first it says no as another PC is logged in. So clearly it is accepting the details and giving a bogus error. So this is clearly connected to the Internet and checking the details. It is just broken.

Clearly cannot code the simplest of "Login" processes if their livelihood depended on it. WTF are Blizzard playing at?

This is proper fraud if you ask me - take my money and give me crap.

Company values

Seems several people posted that @stephenfry should get @aaisp. I can only assume he is pissed off with current ISP.

So I pondered - why would he? Apart from that fact he is smart, can afford us, and is one of the coolest "celebs" I know of...

I was wondering what I really should try and pin down as A&As values.

Obviously, for a start, being open and honest. If we screw up, we say so, and we say what the hell we are doing to stop it happening next time. I cannot see any point not being open and honest. That is how my mind tends to work. We have a clear "no bullshit" policy across the whole company.

We also have a technical angle to the service - doing it right technically. Proper 1500 byte MTU. Proper native IPv6. Delegating reverse DNS for IPs. Heck! Proper IP address assignments to end users. Internet like it was designed.

I would hope our technical support people are all up to the job, but I am sure Stephen can manage the "shibboleet" test if needs be, and be transferred to one of the many people that know more than 2 programming languages.

We also believe in free speech - we just pass packets and have no reason to snoop on them, but have no reason to let .gov.uk snoop on them either. And we have no reason to tinker with, or filter, anyone's packets. Some of the laws on this are plain wrong, or more to the point - some of the way the laws are implemented is wrong.

Having said that, we do register details properly in RIPE for the IP block, and Nominet for domains. We do respect people wishing to have details reduced or concealed if they want, though.

We also expect customers to be honest and up front as well. We don't like being messed about. For the most part all a customer has to do is pay on time. But if there is a problem we want customers to be honest with us. We also want customers to understand what we have promised, and what we have not. We stand by our reputation.

So, I see no reason for @stephenfry not to take our service - and welcome him as we do every one of our valued customers. Happy to answer any questions he has. @TheRealRevK

Reporter on my doorstep!

Fun!

Reporter popped around after my tweet about a murder in Bracknell this morning. Seems they can't read time stamps and thought I had tweeted at 1am, when in fact I tweeted at 9am just after the body was discovered by my daughter's neighbour.

What is moderately impressive if that they managed to come to my home address.

I did not bother to ask, as it is not a secret - one can track me down using domain names, company records, and so on.

But interesting :-)

Anyway, just as well I did not threaten to blow up an airport or something...

Find My Friends

The Find My Friends app is a bit odd. A bit sort of stalking. Very powerful though.

But on hearing someone has been murdered in Bracknell this morning, the first thing I did was hit Find My Friends and see that they were all marked as "Home".

Oddly, Georgina turned down a new iPhone yesterday (yes, I know, odd, but she did manage to destroy quite a few indestructible phones). Shame, as I feel slightly happier that we can check up on the kids, even if that is being a stalker parent.

PGP/GPG Digital Signature credibility

I recently had a discussion on a mailing list regarding digital signatures on emails or files. The discussion was with a lawyer.

As usually happens when techies and lawyers discuss something, they approach the issue from very different directions.

From a techie point of view, a signed email is a signed email, and there are levels of security afforded depending on the key size and algorithm, and so on.

From a legal point of view, signing a document has specific meaning, and somehow you have to confirm that the signer had intent to sign the document, and understood that what they were doing was signing it in the same way as signing a paper document. Signatures using a rubber stamp on paper are, in many cases, valid, but if a document was just routinely signed by a clerk or even automatically signed, then that does not have the same meaning.

So, I was thinking about PGP / GPG signatures. A bit of quick googling did not suggest that what I would like exists, so I am thinking an RFC would be a good idea. Of course, what would be better is if someone can say "yes, it already does that, see RFC xxx"...

Basically, when signing with PGP or GPG you are typically asked to enter a passphrase. This is a clear user interaction and equivalent (in my mind) to signing with a pen. You know you are attaching a personal signature to the document.

So what I would like to see is a tag on the signature to flag the level of user interaction that was deployed to access the secret key and create the signature.

E.g. bit fields for :-

1. Some user interaction (pass phrase) was required to access the secret key
2. User interaction required OTP / two factor authentication
3. Secret key is on a physically secure device that cannot be duplicated
4. A recent cache of the key was used (i.e. no user interaction this time)
5. Biometric validation was used to access the key
6. User chose to positively confirm that they wished to legally sign this (checkbox)
7. A duress procedure was used and duress not indicated

This would allow automated signatures to be distinguished from clearly deliberate signatures, and even give additional credibility to the signature.

The signing code would generally know the answers to these questions and be able to indicate these automatically.

I assume it is possible for new fields to be added to the format for signatures and that these can be the "comprehension not required" type.

Does this sound like a good RFC?

Special offers

Breakfast TV had a big thing this morning on supermarket special offers being misleading. I was quite surprised at how polite the response from the supermarkets were, to be honest.

I have moaned about "special offers" before, the classic being confusion over adding and removing percentages, but this time I am with the supermarkets.

The sort of thing there were slating was a "3 to 2" offer.

This is a very simple concept for everyone to understand, surely. The item has a price, and if you buy one or two of them, that is the item price you pay, but if you buy three of them then you pay as if for the price of only two.

A "3 for 2" offer is not in any way whatsover suggesting that the item price is lower (or higher) than it was yesterday or will be tomorrow. It is a price right now, and a choice of buying a specific quantity for a specific price, or not. It is clear and unambiguous, and not misleading.

The supermarket did exactly what they said - they charged for two item's price when buying three, exactly as the offer.

The complaint, and crux of the TV article, was that "before the offer" the item price was lower than when they started the offer. In this instance the "offer" did make the price per item lower than before when buying three items, but buying one or two cost more than before.

BUT THE SUPERMARKET NEVER SAID THIS WAS A CHEAPER PRICE THAN BEFORE!

They were not at all misleading in their offer. Prices can change whenever they want and by whatever amount they want. There are rules on claiming a "reduced price" to avoid prices going up for a day and then going down a bit, etc. A "reduced price" offer would be misleading if it was not indeed a genuine reduction. But this is not one of those instances. It was not misleading. It was "3 for 2" as stated.

If the TV article had been at all balanced they would have pointed this out, but no, they claimed it was misleading, and even managed to get the supermarkets to apologise for a "mistake"...

Madness.

Diablo 0

So, thought I would try Diablo 3.

Paid for it.

Downloaded it.

Ran it.

Error 315300.

Followed all the tricks on the fora, and no joy.

Crap.

That is all.

Holiday

OK, someone has got to 40 and I am 31 (this is on a PvP realm this time), so I think I need to go on holiday to Azeroth for the day...

Which leaves me pondering the holiday we have booked.

Given that it was somewhere hot (Greece) my only real stipulation was that it has air-con, Internet and alcohol. Apparently what we have booked meets all three criteria.

Now, do I try and engineer playable WoW machines of some sort as well - that is the question. I believe there are some laptops designed for serious gaming (and no, I am not a serious gamer, you know that)... I'll have to investigate.

P.S. have you ever tried telling Siri to send "level 40" in a message? It sends "level XL". I wonder why Siri thinks "level" prefixing a number should convert to Roman numerals. Odd.

Update: I got to 37, but leveled my cooking a bit as well, and my son insisted I take him to dinner at the Warren. Now chasing level 41.

A new ADR

So, we are finally on CISAS. There are only two approved ADR companies, and this is the other one.

I have to assume they are the lesser of two evils as it is hard to conceive that they could be worse.

The terms are slightly different, and generally seem better. Of course, the terms of the last bunch seem to have been totally ignored by them. So it is possible this bunch will do the same. Lets hope not.

I really hope we never have to find out.

One small detail they seem to have is that they do outlaw claims where they are subject to proceedings in the courts. The last lot did this too, but only if the customer had started the proceedings. This lot seem to not have that caveat. This means that if we have a dispute with a customer not paying, then starting court proceedings will stop them using ADR against us. This makes sense, as ADR cannot rule (enforceably) in our favour, so no point in ADR tackling such a case (you would think). The last bunch seemed happy to take what was, essentially, a customer not paying their bill, agree with us that we are not in breach of contract, and then insist we waive the charges and pay them money, which is blatantly batshit crazy (IMHO). If ever we have a new case we will be very quick, up front, to quote T&Cs to the ADR and, if needs be, argue with OFCOM.

Anyway, we are asking a few up front questions of CISAS to clarify a few things just in case. I hope they are OK on this. If we have a case that goes to an arbitrator then something has gone very wrong.

But it does get potentially interesting. The contract with the previous bunch does not end until next April, AFAIK.

However, my understanding, which may be wrong, is that one cannot simply take a dispute to an arbitrator. It basically has to be in the agreed contract that you can use an arbitrator in the first place.

In the case of telcos, OFCOM insist that this is the case (as required by the Communications Act). So we have that option in our contract terms.

We have now changed our terms so they say that eligible customers can (following the customer complaints code, etc) take a dispute to CISAS and not to Otelo.

I think that means Otelo cannot take on any more cases for our customers, even though we are still in a contract with them. I do hope so.

It also means that our contract can, correctly, only allow ADR for people the Communications Act says can take us to ADR. This is subtly different to what the scheme rules are for CISAS. It is subtle. CISAS say, for example, a company of more than 10 employees cannot use ADR. The Comms Act says "for which more than ten individuals work (whether as employees or volunteers or otherwise)". not a big issue I expect, but surprised they do not align. Now, if I am right, and the contract terms have to allow arbitration in the first place, then this is fine. Our T&Cs limit to people as per Comms Act, and then people have to meet scheme rules. We'll see what they say.

Please don't get me wrong. I want to be fair and reasonable with all customers and want to resolve any disputes promptly. We *do* have a contractually agreed, up front, in your face, limit of liability, and we don't expect anyone to try and bypass that. We certainly don't expect to have to pay some arbitrator to resolve a dispute for us. One case in 15 years is not a bad record, but that should not have happened in so many ways.

Dragging Pauline in to the 21st Century

Nice fast BT engineer, 17 minutes down time, and changed from 3Mb/s to 80Mb/s at Pauline's house.

"Is that good?" she asked.

This FTTC is pretty good really.

That said, we have a 330Mb/s customer on line today (FTTP) which is just getting silly.

Pushing down the list

Is it me? or is the picture of "nanny Dolly" on my blog freaking anyone else out?

I have to make another blog post just to push it down off the screen...

:-)

(P.S. that worked, on my screen)

Pushing boundaries

We are hitting limits on WBC links at present, typically around 10pm, and have more bandwidth coming on-line to BT at 11pm tonight.

Looks like we are well over a gigabit with BT and BE now, which is quite scary. I expect we will put more on next month and plan is to get even more before the Olympics start, and then see how it goes.

It is amazing how this has grown, from our first 2Mb/s BT link in Reading, the 8Mb/s and 34Mb/s BT Centrals. It seems only yesterday we had two 155Mb/s BT Central links, and now look at us!

I am pleased to say the FireBricks are scaling nicely to meet the challenge, and it is nice to see more and more small ISPs taking them on instead of the established alternatives. I should we worried, as it means A&A losing our edge, but to be honest, the more ISPs that can keep BT in line with our constant quality monitoring, the better. It is better for our customers overall, and that is what matters.

A&A would not be where it is today without FireBrick!

Tiring

Well, another 2 day FireBrick course completed.

6 people on the course - I hope they liked it.

Why is running a course so damn knackering?

All change

So, I changed the layout of the blog, and some people do not like it.

I am not doing well changing it back, but I'll see what I can do!

... OK, this is more the original style, is that better?

RIP Dorothy

Well, how do social taboos work? I assume that taking the 1DsMkIII and 70-200mm lens to a funeral is a no-no, yet to a wedding or a christening it is fine...

What is strange is that camera phones are not an issue, and pictures of the flowers, and so on, are all OK. Indeed, for reasons I will not go in to, it seems several people posted pictures on facebook of happy smiling faces of many people at the wake. You would think it was a party, if not for the suits and black ties.

Strange how some things go, but was the least cringeworthy funeral I have attended and I have to say very well done to Sandra and Pauline for organising it - they had a fine line in compromise with grandchildren having fond memories, and some others perhaps less fond. Cudos on the music, and well done.

Regardless, the loss of anyone is sad - R.I.P. Dorothy.

Welcome to the future!

I have had an iPad for a while, but for some time now I have used a sonim as my phone. They are nice, and (except for teenage daughters) indestructible. However the microphone was getting iffy and so I went for a new phone.

I got an iPhone 4S, which is, I know, turning to the shiny side. It does seem to work as a phone, which is good.

The iCloud sync to the iPad is nice - I add new contacts to the iPad as the keyboard is more usable, and they appear on the phone. That is indeed cool!

I am however reminded of how much things have changed - even just during my lifetime. There have been authors writing about the future for a long time, whether Jules Verne, H G Wells, or Gene Roddenberry, they have envisioned the possibilities, often with no consideration to the laws of physics. And whilst some of their ideas are just too fanciful, we are seeing so many of them come to life in the last few decades.

I feel luck that I grew up just as the latest explosion of invention and innovation took off. I think the late 1800's were similar in some ways - an exciting time to live, but the last 30 years or so have been amazing.

When I was at school we had the very first home computers. Atoms, TRS-80s, and even ZX80s. I recall cycling to school with a TRS-80 and a b/w TV on the back of my bicycle on more than one occasion. We had no mobile phones - but I did have a CB radio. We had no Internet. I recall a TV series about one of the first space policemen, and he had some experimental device "box" that allowed him to just ask it to get information. The iPhone just does that, and a fraction of the size. At the time it was pure science fiction, not just in the communications technology, but the information technology, both of which teenagers just take for granted now.

But really, the likes of an iPhone are just so far beyond the wildest dreams of those days, even science fictions writers could not imagine.

One thing that is amusing is old star trek episodes. They envisage matter transporters and communicators, but still had switches and buttons. Back in those days the controls of anything were the buttons, levers and switches that made things happen directly, or via simple electrical connections. They could not envisage the separation of control from action we enjoy today. The idea of a keyboard or mouse was beyond them as they could not see what they would do. Yet even now, we are seeing the demise of the keyboard and mouse for the intuitive touch screen, voice recognition (which they did predict) and gesture controls. We live in a changing world.

I imagine, as voice understanding (not just recognition) develops over time, we may see the keyboard and mouse disappear, a transient stage between "real" switches and buttons, and computers that really understand us. The idea of typing will become a relic of history.

For years the idea of video calling was science fiction, but it is happening now. I spent some time on a video call the other day discussing how, if that had been a "real" phone call, it would have been much more expensive than the video call!

So, the business cards now have iMessage and FaceTime icons on them.

Welcome to the future! [Philip J Fry, 3000AD, Applied Cryogenics]

Meeting the standard - come on ISOC!

World IPv6 launch is very soon - a few weeks away, but if you look at the ISOC web site you see just two equipment vendors.

Yet there are some DSL CPE manufacturers out there who are really trying. Technicolor for one, but we also have Zoom and Billion with working routers, and others claiming to be working on IPv6 CPE. As an ISP we are shipping the techicolors and they are working (as well as any router does, i.e. with the odd quirk, but working with IPv6, thank you very much).

So why so few equipment vendors? Why not even FireBrick listed?

After all, the current range of FireBrick products has been developed with IPv6 from the start as part of the operating system design and every other level of networking and application. We do IPv6 as well as we do IPv4, so why are we not listed.

The problem is that for any equipment manufacturer, it is almost impossible to be 100% RFC compliant or pass a barrage of strict tests on the RFCs. This is the case for IPv4 and IPv6. I bet most equipment our there does not meet fully a complete compliance test for IPv4. And to be honest, do they need to? If they work well enough to be usable by almost all customers, and they are prepared to take bugs seriously, is that not good enough?

The view we have in FireBrick is that we want to be standards compliant as much as possible, but also to make a product that works well and does the job. If someone comes along and says we do not meet some RFC, and especially if they actually have a good reason why they need that (but even if they don't, in most cases) we'll upgrade the s/w (free of charge) to meet the RFC. There are some cases where we deliberately have options to work differently to the RFC, e.g. we were one of the first BGP routers to have an option to ignore badly formed transient attributes after bugs took down large parts of the Internet (since then,  this is now an RFC I believe). Where an RFC is vague, we work with customers to make sure we have the best way of working. We even have compliance options with some of the April fool's RFCs! This flexibility is key to making the FireBrick a swiss army knife of network appliances and allows a lot of clever stuff.

So, do we pass the formal testing that ISOC want for world IPv6 launch. Probably not 100%, to be honest, no yet. We may be inclined to give it a go. But do any of the home routers people use now pass the same level of testing for their IPv4 stack? I seriously doubt it - but that does not make them unusable.

I would hope that ISOC can consider listing vendors that have working implementations (as self declared) and a stated commitment to IPv6 support. This would cover many vendors that are taking IPv6 seriously, including FireBrick.

Come on ISOC - add us to the list please...

Adrian's apple

Well, I went for a very personalised iPad.

It has my brain on it. You don't get much more personal than that.

It is a bit weird, I know.

But it is pretty unique.

As some have commented, it is an odd place for an Adam's apple, but you know I am not conventional.

Postal fraud or just crazy person?

Well, someone I know was messed about by a landlady, as happens, and so she sent a letter asking the landlady to pay for some items she disposed off without giving my friend a chance to collect them. Long story, and I am not going in to details.

The landlady writes back, and amongst a lot of seemingly crazy things in the letter, she points out that my friend put a recorded delivery sticker on her letter. She says that as she did not sign for it, it was not recorded delivery, and so putting that sticker on the letter is fraud. Apparently she is going to report this fraud to the police.

I am at a loss. Obviously, it was recorded delivery - that is what the royal mail recorded delivery sticker means. Clearly the post man simply forget to get a signature (as does occasionally happen).

Interestingly, she is not claiming that there was insufficient postage on the letter or that she had to pay any excess postage, simply that saying it is recorded delivery when clearly it is not (as she did not have to sign) is fraud!

I have suggested that having this (apparently) crazy person in the same room in front a judge is probably going to end badly and not really worth the hassle. She is already making various wild accusations (including the post fraud) and my friend cannot prove the value of what she disposed of. Shame, but some times it is best to just walk away.

I almost hope she does go to the police with this apparent postal fraud though. Shame we would never hear how that goes.

There are some "interesting" people in the world, aren't there...

P.S. For added amusement - we have just checked, and Royal Mail have a proof of delivery signature for the letter...

Clueless B2B XML

OK, I have a lot on my plate, and a funeral does not help matters, but I am working out what to do over the next few weeks.

We have the VoIP work to progress, but what we have now is a good milestone of "usable PABX", and the next stage is all the integration with back end systems to make a large scale telco SIP router. Its a long term plan. I also have OSPF, just in case you thought I had forgotten.

However, there is increasingly a need to progress the "clueless B2B XML"...

WTF is that? you ask?

Well, we have a management system for DSL, FTTC, ethernet, domains, email, all sorts. It runs on clueless. It can link in to the accounting system (priceless) as well.

So the plan is to make a new XML interface. And seriously, no, I am not going to do SOAP. It is a waste of space. However, I have conceded to use XML.

The plan is that, with suitable username and password, you will be able to extract an XML definition of all services you have for an account or login.

Then you will be able to make changes to that, and send it back.

The back end will work out what the changes mean, and provide details. These will be URLs to fetch for a PDF or XML order confirmation, a text description of the changes, and also the charge for the changes (or credit) if applicable. There will be a confirmation URL. You can then use the confirmation URL to agree charges and request the changes and hence get the invoice URL, if applicable.

The idea is that a load of non-chargeable changes, like changing contact details, passwords, requesting interleaving on a line, etc, etc, can be done easily.

But if there are any charges, you have to have an account login, and then confirm the charges which will be invoiced.

This would allow changes to existing services, ordering new services, ceasing existing services. A special case is a new customer which means sending a bare XML in the first place rather than modifying the existing services XML and sending back.

Then we write a few front ends using javascript and the like to allow simple orders (new customer with single broadband) and more complex orders (multiple lines, etc).

It will allow dealers to manage their lines, and place orders. It will allow us to have more than one type of order form for normal customers. It should be good.

It may be started with some of the basic services and extended over time. I plan to have a published XSD for it as well.

Comments?

The problem with Communications Data and the Internet

"Draft Communications Bill - Will allow the police and intelligence agencies to collect data on communications, like texts and emails, flexible to changes in technology, such as the internet. This will apply UK wide."

Whats wrong with that then?

The history of this is really to do with telephones. As soon as telephone exchanges were digital the telco (post office / BT) were able to get data on calls made. This was used for billing, and disputes, and so on. It was, of course, invaluable for the authorities to be able to interrogate this communications data to find who called who and when. This helps with investigations of all sorts of crimes.

Now the Internet comes along, and suddenly they realise they no longer have this. People are sending emails, and making telephone calls, and all sorts, and they don't have anyone to ask for the communications data.

They have gone some way to try and tackle this with the Data Retention Directive, which means anyone providing telephone and email services have to hold the data they are processing for a year (if asked to). But even that is rather out of date now in concept and only applies to data they already process. It is also (IMHO) very badly drafted.

So why is it so hard exactly?

Is the data there even?: With telephone calls there was a need to collect data in the first place, for billing. With email, and messaging services, and even some VoIP services, there is no need to collect the data for billing. It does not matter how many messages I send on irc for billing.

Who collects the data?: With many services there is a telco providing the service itself. E.g. a traditional voice call goes via the telephone exchange. That means there is someone with the data. However, anyone can run an email server and they don't have to be a telco, or easy to trace, or subject to logging requirements. People can run their own email server at home, and send email to other people running their own email server at home, and there be no mail server in between. The lack of central service provider in communications is increasing with more and more means to communicate directly (as Internet Protocol intended). This is even easier with IPv6 removing NAT from the equation. There are peer to peer protocols that are specifically designed around the principle of no central control or authority (for all sorts of good reasons) where there is no service provider at all, and this will happen more and more.

Where is the service provider?: Even when a service has a "provider" who will be able to collect communications data, they could be anywhere in the world. They do not need a presence in the UK or be subject to UK data retention or investigation laws. This is commonly the case even when people are not trying to avoid the legislation.

Broadcast data: One interesting thing about the likes of twitter, or usenet, is that the data is typically sent to everyone. There is no way to identify a recipient. With systems like usenet you can make such messages private using encryption. With no need to reply to the specific sender, the postings can be anonymous. So the communications data becomes "anonymous posted to public forum", but no record of what they posted as that is content.

Can you see the data: One idea is that ISPs could have to deep packet inspect communications (something not even allowed under EU law AFAIK), to extract communications data that is part of non-UK services. But computers are now fast enough for encryption to be completely standard in many services - so no way to actually get at that data in the middle.

Too many protocols: When there were only phone calls, it was easy. Even if it is only calls and email and allowing for VoIP, not too hard. But there are millions of ways to communicate, with and without there being some service provider. In-game chat is a classic, and applies to everything from world of warcraft to wordfeud and there are new apps, and games every day, many of which may happen to provide a means to communicate. There is no way any interception systems could keep up. Even knowing the system, e.g. facebook, the web interface the DPI is trying to track will change at the whim of the designers. Anyone that have tried to screen scrape such systems will know it is a big job to track this. What is worse is, that in order to try and keep up, the black boxes would have to have remote administration independent of the ISP, and that allows a lot more interception to be done without anyone knowing what is going on.

Micro telcos: As I say, people can run their own mail servers at home, but there is a level between that and traditional telco where someone runs services for other people, for money or not. This is so easy now. I used to run a mail server at my house for my family. If such micro-telcos are to have a burden of collecting, storing and reporting this communications data, that would be horrendous. If running an irc server meant keeping logs, that is a burden. If the tax payer has to pay for the black boxes in every bedroom ISP, that is expensive. If it becomes know that small ISPs are exempt, then where do the bad people get their Internet I wonder?

Fine lines: With traditional telephone calls, as a side effect of why it was collected (for billing), the communications data is simple - date/time, from number, to number, duration. That is about it. But what constitutes communications data for email, or twitter, or MSN, or irc? Is the subject of an email included? Is your friends list on facebook communications data? What about all the other useful headers in email? Maybe the ID of a PGP signature used? Where exactly is the line drawn? Even an email address may include someone's name - which is not something that was in traditional telephone call logs. Does the IP address of every IP packet count as communications data? This is an tricky questions.

Signal to noise ratio: With traditional telephone calls, the integrity of the data was good. CLI was only from trusted sources. You were sure the call from and to numbers you had logged were right, with very little chance of error or deception. Now, even on phone calls, you cannot trust the CLI. You certainly cannot trust email addresses. And of course, the vast quantity of junk mail out there - when separated from the content that spam filters use to identify it, it will make for huge amounts of noise. You certainly would not be able to use a record of an email captured from such a system in court as the defendant could point to thousands of emails which are bogus. Indeed, it would be in people's interest to have a virus on their machine sending lots of email to random addresses as it gives them plausible deniability. Of course, if people want, they can make apps that generate low levels of traffic that look like communications - e.g. apparently sending 10 small emails a second 24, hours a day, from a million of computers, to and from random addresses. In there somewhere is the email you are looking for.

There will always be ways to hide communications: All of the above is before you consider someone actually trying to hide there communications. When you start using encryption, steganography, vpns, tor, and so on, then you are able to communicate with no trail being left. So, only law abiding innocent people will be affected by this - the criminals don't have to have their privacy invaded. Are the criminals smart enough? Well, there are plenty of web sites explaining to people in China and other oppressive regimes how to bypass monitoring and firewalls - so anyone with access to google is smart enough.

Consequences: It is all very well saying that this is a total waste of time, for all of the above reasons, but is that a reason not to do it? Well, obviously, if it costs public money, then yes. But what else could go wrong. This data is valuable, and a target to be stolen or unscrupulously sold. It is an invasion of privacy. It is technical complication making things break more often. It could allow general purpose unsupervised black boxes in to ISPs with no end of possible feature creep. It will cost a fortune and so put up prices or taxes for us all.

Someone please educate the politicians!

May the geek be with you

Well, it comes to something when I ask for help with a computer!

But to be honest, the progress of linux distros towards unusable GUIs has made leaps and bounds. I am finding it almost as painful as trying to use a windoze machine some times.

ubuntu wanted to upgrade - so it asked me 50 times in one window on top of another! I closed 49 of them and said yes only to be screwed over and broken.

Thanks to Jimi (one of our tech guys) for just sorting it for me. All done remotely, and not even during work hours.

It is not that I couldn't do it, really, just I know a man that can sort it way faster than I. Jimi is the man. Well done.

One of the magic incantations was :-

sudo apt-get remove unity

Another was:-

sudo apt-get install tcsh

How he managed to persuade it to actually finish the upgrade cleanly enough to do either, he will have to let me know some time...

Is the telephone call dead?

We had a rather bizarre conversation last weekend, which made me ponder how long people will actual use telephone calls.

Basically, it was down to planning what was going to happen the next day, and who was meeting up where and co-ordinating luggage and people getting to the and from the right place at the right time to fly back to Sweden after spending much of the day in London.

Basically, various communications methods were discussed, including the "chat" on wordfeud. At one point it was suggested lending an iPad so they can use iMessage and Find my Friends.

It was after several minutes of discussion that finally the idea of actually phoning each other was considered. This was very much as a last resort. In fact, nobody knew each other's phone numbers as they had never had cause to use actual phone calls. We had emailed, imessaged, chatted in games, even facetime'd, but not actually used a phone call.

I'm glad to say that face to face talking is not dead yet, but phone calls seem to be on the way out. That said, it is a tad strange having a room full of people playing scrabble on iThings with each other while the TV is on. What is the world coming to?

Sky1 "Touch": Irresponsible insulin injection!

There is a new series on Sky1 at the moment, call "Touch". As a series it is not bad, very watchable. It happened to be on TV last night.

During the episode the main character (Kiefer Sutherland) finds someone unconscious on the floor. He called emergency services - good - right thing to do.

But then he used the man's phone to call one of his friends, who then said "He is diabetic" and directed him to get insulin from the fridge and inject the man - within seconds he woke up and was well - hero!

This is seriously irresponsible. This is prime time TV and watched by a lot of people.

My daughter already had this misconception. She was likening my insulin pen to an epipen used to treat those with sever allergic reactions in an emergency. She had seen that on TV.

Diabetes means that your body does not produce enough insulin, and (once medications are no longer effective) a diabetic will take insulin. A diabetic taking insulin has two main problems because we are using injections to replace what a healthy body does normally:-

1. High blood sugar. This can happen if we forget to take insulin or don't take enough, or even just eat too much. This makes you thirsty, tired, even sleepy. You don't feel too well. It is not good for you in the long term, and untreated diabetics can suffer a number of nasty side effects of long term high blood sugar after many years. Eventually it will kill you. It does not make you pass out or put you in a coma!

2. Low blood sugar. This happens if you take too much insulin, or simply don't eat enough. It makes you irritable and disorientated. It can be confused with being drunk even. It can cause you to pass out and go in to a diabetic coma. This is bad, very bad, and can kill you quickly (within hours).

So, what do you do if you find a diabetic that is passed out? The last thing you do is give them insulin! If you do - you are probably murdering them.

The best advice is call emergency services. If you know the person is diabetic, tell them. They can advise you if there is anything you can do.

Generally, if you can wake them, getting them to eat or drink something with sugar in it is probably going to help. One issue with a diabetic that has been drinking is that they seem drunk - and that is usually "treated" by letting them sleep it off. That is not so good if they have low blood sugar and can mean they never wake up.

I am not a doctor, just a diabetic. But please: don't try administering drugs because you saw it on a TV show.

Of course, on the TV show, the character is given an injection of insulin in the arm with no thought to what dose to give, or which sort of insulin it is. I currently take a slow acting insulin that takes effect over around 24 hours. Even if I did have high blood sugar, an insulin injection of that type would not have an instant effect. But even if I had high blood sugar, a large dose of fast acting insulin could cause the opposite effect quite quickly and put me in to a coma.

There are enough people with diabetes around who should have been able to spot this silly mistake during the shows production. Why was this aired like this?

Blocking the pirate bay

It seems the big ISPs are ordered to block the pirate bay in the same way as BT was. It is silly. Most blocks like this can be bypassed simply by using https, but there are always going to be many ways around the block, and even if the block worked research already shows that it has no actual benefit in stopping downloads generally, so is simply not appropriate as a legal tool.

People have asked, and no, AAISP still do not filter anything and have no plans to.

The problem is that things have moved on so much since betamax, VCR, or printing presses, that any form of electronic communication (even tunnelling over DNS) could be used to communicate with unrestricted and even encrypted data. That means you provide anything whatsoever by way of communications and you allow everything.

So, restrictions on electronic communications can only ever be effective if we break the whole communications system so badly it is not usable.

Everything else is partly effective at best. Of course knowing how to bypass things used to be hard too - but if you have access to "the Internet" and "a computer" you can find the tools and instructions to bypass things easily.

This means if people want to communicate then they will be able to.

The only fix is to stop them being motivated to communicate. That means a business model for things like copyright that make it simpler, and easier and cheaper for people to do the right thing. Right now, even people that have a strong bias to do the right thing, and the means and desire to pay for things, find they cannot do so because of the chosen business models of the copyright holders. When you have that scenario, not only will people communicate, but there is suddenly a black market where people can charge money to make things easy - money that should, and could, go to the artists if only they would try to work with their customers.

Copyright law could change - it could be (as is some countries, I belive) that non profit personal copying is always allowed, and that only commercial exploitation (which has a money trail to follow and enforce) is a problem. That would avoid the need to do anything at the ISP level, just follow the money!

This whole issue is very topical as well when considering the calls for filters on porn ("think of the children", yet again).

A filter can work if its objective is to help people avoid seeing something by mistake (IWF blocking for example) - where the users are actually keen to have the block. This applies to thinks like blocking "malware" as well, and is why many ISPs do offer spam and virus filtering stuff. The users want it, and will even pay for it, so the blocking works.

The second you have someone with motivation to work around the blocking, it will not work. You make it a tad harder, maybe, and that is all. You don't change behaviour. You can't block people motivated to find porn on the internet (e.g. teenage boys). You can't block people motivated to find music and videos that they want. The best you can do is mask the issue.

I think we have to campaign for a clear separation of responsibility. Just like the post office is not expected to open every letter (whether technically possible or not), we should not be expected to police or inspect or filter internet access. There is no point, and there is a lot of harm in trying.

Even suggesting the likes of facebook are legally liable for what is on their site, as some suggest, would make facebook impossible as a service. There is no doubt that services like facebook and google and so on are there because of the freedom the internet currently offers. We would stifle that sort of innovation and business growth if we make a "locked down" internet, whatever our motives.

That does not stop ISPs offering services that do filter some things in some ways or applications on PCs, but we should not be forced to look at a single packet.

Not only are there the whole cost; technical; and pointlessness issues, but we have the whole "free speech" and "thin end of the wedge" issues.

I hate to say "I told you so", but when there was campaigning for 100% IWF block list take up by ISPs, I spoke out against it. I said it would quickly expand beyond its remit. Now we see that, not just within the IWF, but by the courts using the monster we have created. We see the easy targets first - porn, copyright, terrorism. But you can push it one step further every time until you are banning free speech and creating a repressive state. It has been seen so many times before and the only way to tackle it is at the start - not to allow that thin end of the wedge and to suffer the comparatively minor inconvenience that results.