Decisions

I have a new iPad, duh, obviously...

The old one had an orc engraved on it (see picture).

So what do I put on the new one.

Need to dream up something before going to UKNOF, else reputation trashed and could be a tad orcward.

I have done stargates before. One of my customers has some space ship on his (at the AAISPISSUP - alcohol and lasers do mix!)

And the orc is nice, but same as last time, so something new.

Hmmm, decisions... decisions...

Odd day

OK, after an odd weekend, an oddly good day.

Firstly I would like to thank the people that have called and emailed - close family are complicated at the best of times, and things are more so when one of them dies. Looks like it is all in hand, though a coroner is needed as it was rather sudden. Lots of mixed feelings by many people and the occasional cross word here and there. Funeral all arranged.

And Pauline, no, I am not adopting an orphan!

But good news on bug hunting - found two buggers today.

One was to do with RADIUS accounting and clock setting on the LNSs, and was, as ever, a really annoying one-liner which I introduced recently. It was a fix to another bug I had introduced a few days before. So we're rolling out s/w upgrades to LNSs over the next few days.

The other was VoIP related (phone system in office). I had looked for ages, but the extra debug finally helped. Well, I assume it is fixed, it fits the bill. It involved a call to a hunt group ringing multiple phones and one of the phones answered (not the first one in the ring list) and then later transferred to another group, and I think even that had to be transferred on as well, and then the call had to last while another call came in to the middle hunt group. Some times I hate linked lists, and have added one line of code that was missing. Arrrrg.

On top of that we had a good discussion with the other favorite telco over the BE lines, and are in agreement that we want to work together to make sure faults get fixed properly. All is good.

Now for staff reviews, and UKNOF, and Reading beer festival - long week ahead.

Surreal Weekend

Not sure what to say about this weekend...

My son's girlfriend's parents came over from Sweden, so we were entertaining them. Nice people. Fun weekend. Went to see We Will Rock You at the Dominion, which was quite impressive. I hope they enjoyed it.

Not helped by my mother-in-law unexpectedly dropping dead on Saturday afternoon. This is getting a tad close for comfort - clearly I am getting old.

So, a busy week ahead helping my wife wherever I can with organising things.

Bug hunting

Arrrg!

OK, enbugging code is the process of adding bugs to it.

Sadly it is almost impossible to do "coding" without also "enbugging".

I am chasing a couple of annoying sods right now. As ever, they are "impossible" until viewed in hindsight. If only I could have hindsight, in advance :-)

The VoIP code has an issue where, every few days, in the office, the support hunt group is breaking. We can see it happen and reset the call server in the office when it does, but I cannot see any way for this to occur when reading the code.

I am adding more and more debug logging to the code - I will find it.

But bugs you cannot reproduce on demand are the worst, and the ones that take days or weeks to "just happen" are even worse. The whole cycle of finding likely causes and adding more debug information takes weeks if you are not careful.

Well, all I can do is wait and see.

Oh, and if you wonder what is worse than that - it is compiler bugs. They are very very very rare, but I have hit a couple in my many years coding and they are the worst type of bugs you can imagine. Trust the compiler. Use the source, Luke...

EEP to Nominet

We wanted to make the handling of UK domains a tad slicker in our systems, and in particular allow end users to update details including name server and DS records.

Fortunately Nominet have EPP (Extensible Provisioning Protocol) which allows some rather neat automation.


Well, I was half expecting it to be some SOAP/XML thing, like so many other XML based APIs, and was pleasantly surprised that it is not. It is just XML "messages" each way sent with a length (in binary) and the XML ,over a TCP (or TCP+SSL) connection. Simples.


I have taken just over a day to code all of the functions and integrate in to our management pages. It is damn quick too - registering domains instantly. We used to use the email based system before, which worked, but at the speed of email.


So, finally customers can set up DS records. All we need now is signing domains for people. That will come later.


One rather annoying thing, which is not an issue with EPP as such but with XSD, is the definition. It insists the fields are in order. Why the hell is it done like that!!!


Basically, a "structure" can have two styles, either a strictly ordered sequence where individual items may appear a different number of times (you define min/max number of times), or you can set any order but only a maximum of one instance of each record.


What we want is any order and defined number of instances of each entry.


As an aside, that is what we do with FireBrick config. It is generated in strict order as per the definition, but accepts in any order. Much easier.


Sadly the EPP requires the strict order, which is just a nuisance and so unnecessary, and what is worse is that if you get it wrong the error is just that there is a syntax error (as it fails their xsd validation). No clue what bit it does not like. Shame.


However, good fun, very easy to code if you ever want to do EPP yourself.

Is it fraud?

Surprisingly facebook seem to be promoting people that sell WoW Gold.

I was surprised, and suggested that surely this is promoting some sort of fraud.

This sparked a bit of a debate - is selling WoW gold fraud?

My view is that they are claiming to do something which can't be done without the breaching contract with WoW.

Reading the T&Cs again, it looks like selling WoW gold is the breach of T&Cs not buying it, though I a may be wrong there and it may be both. That said, you cannot get what you pay for when you buy WoW gold as there is no actual transfer of title to the virtual gold. So surely someone "selling" WoW gold is being fraudulant? Are they?

Even so, it is clearly a dodgy business, so surprised facebook promote it.

Nanny state - think of the children

This is getting slightly out of hand.

There is a private members bill that has not, apparently, been thrown out (On line Safety). As it stands it is just very very broken as a bill or a law. But the whole idea is flawed really badly.

Basically the idea is that some politicians mistakenly think the Internet is like TV and magazines - a content service. It is not, it is a communications service, like the post office or telephones.

We do not have any regulations that says BT have to censor certain words said on a phone call. We have no regulations saying Royal Mail have to check all post and censor certain words or images being posted. That would be mad.

Somehow, some people, think it right that we need an Internet where ISPs have to somehow censor "images" being accessed.

Tell your ISP what level of filtering you want?

Technical issues

Firstly, as this is what I do best, the technical issues.

The bill calls for ISPs to offer a service which excludes porn images. ISPs move packets. We do not provide content. Asking us to exclude porn images is crazy. As an analogy, it is like expecting manufacturers of glasses to make a system to stop people looking at porn magazines. Technically, with an LCD shutter, GPS, accelerometer, giro (all the things found in an iPad) and a database of the location of all porn magazines on newsagents shelves, etc, it could be done, but it is the wrong way to do it!

No ISP could offer a service with "general access to Internet" and "no porn images". It is simply impossible. There are so many ways around it. Even if an ISP offered no more than DNS lookups, there are systems in place to allow IP over DNS tunneling and so general Internet access via that (used to break "walled gardens" and "pay gates" for no fee). Much more simply are remote proxies, and https porn sites, and encrypted emails and so on.

Allow anyone to send a 1 and a 0 and you have a means to send porn images.

Don't get me wrong, there are ways for ISPs to offer some level of parental control. By no means fool proof, but something. That said, this bill seems to try and circumvent the resourcefulness of teenage boys trying to access porn with all the access to google and facebook they need. Not going to happen. But no problem with ISPs trying to offer some such services if there is a market for it. Some do, and good luck to them. We (AAISP) don't but you can install apps on your PC.

You can do slightly better with applications on the PC to manage parental controls, and many free and paid for apps exist for that. So why force anything on ISPs? The "solution" already exists...

Side effects

Another aspect is the side effects. It would not be quite so bad if ISPs were expected to simply "explain the filtering services they offer". But the wording of the bill requires ISPs to ask people if they want to "opt in" to "porn". They then have to intercept and monitor their customers traffic (legal?) to check if they are accessing porn or not.

There are technical issues, as have been seen with IWF based filtering. Basically, systems designed to stop people accidentally accessing kiddie porn have been deployed by many ISPs. Well done to them, but these systems were never aimed at stopping access to the material, just ensuring people don't hit it by accident. Even so, such systems have caused serious issues with sites like wikipedia because of the way they work technically. Similar systems to block general porn will be equally ineffective against someone trying to access it, and equally disruptive to "normal usage" of such sites as wikipedia.

There is also the fact that you create nice lists of sensitive information such as "who wants to access porn".

On top of that the logic is binary. If someone in a household wants to access porn, as is their right, then the ISP is not filtering the whole house!

Is there actually a problem to solve?

Porn is a fact of life. I agree, some people find it offensive. Some find it degrading. Kids, especially teenage boys, will find a way to access it. This is no different with the Internet than it was 20 years ago, just you don't have to try and find porn mags under your parents bed any more. In my opinion it is better that kids are educated that it is fiction, and not how one should treat people in real life. They need to understand, as we all do for all fiction on TV, that there is a massive difference between fiction and reality, else we would be banning a lot more films from TV and cinema! Why is porn any different to Lethal Weapon or Saw III. It is far from reality but is entertainment for people that like that genre of film.

Find me a man that has never seen any porn? Did it cause them serious psychological harm? Make it "not allowed" and you bypass the education that needs to go with it as it is no longer discussed.

Thin edge of the wedge

Many ISPs filter kiddie porn, fair enough. Next filter "normal porn". Next filter "terrorist web sites". Next filter "politically incorrect web sites". Next filter "wrong thinking". Then they came for me...

Sorry, but freedom of speech is too important, and allowing any inroads in to that is the thin edge of the wedge as seen so many times in the past. Learn from history. Put up with some unpleasant stuff in the name of anti oppression.

Parenting

Parents - be parents - educate kids on difference between fiction and reality - educate them on using the Internet wisely. Supervise them. Do not expect the government to pad them with cotton wool!

I have 5 kids, by the way.

Picking IPv6 addresses

How does one pick an IPv6 address for a machine?

Whilst IPv6 deployment will usually "just work", and may even just be handled by a broadband router and an ISP that do IPv6 (like AAISP), if you are setting up a network (e.g. large office, or multiple sites, etc) you have to consider things a tad more carefully.

This is especially important for an ISP like us, as we have to be able to understand the IPv6 addresses themselves if trying to debug the network when things break (and one thing that can break is DNS, or access to it).

Side track: We have had a couple of cases recently with support where a customer has had some issue which means he cannot get on-line properly. Usually with their machine or network. However, they have got on irc to ask for help! This is because the IPv6 just works. Always amusing someone using the Internet to ask for help because the Internet is not working.

Numbering the LAN

Each LAN will be a /64, that is pretty much defacto now. As an ISP we allocate people a /48 to deploy as they wish. If you have many sites you probably want to split the /48 in to parts, e.g.  a /56 per site allowing 256 sites, and 256 LANs per site. This will depend a lot on how you manage sites now. If you have any sort of site numbering, you could perhaps use that as part of the address.

If the sites need independent BGP announced routing you will want to consider additional /48's. It seems you can get a /48 to propagate around the Internet to some extent. As long as someone announcing something bigger (e.g. a /32) can see all your /48's then the /48's themselves do not have to go everywhere in the Internet. It is also not nice to make lots of separate /48s in BGP if you can avoid it. As with IPv4 routing there is no guarantee of routeability of any block so you need to play nice if you want things to work.

This weekend we changed our office allocation to a /47, i.e. two adjacent /48s. This allows a single filter in access lists, but allows one /48 at each of the two independent sites we run, though we have the /32 from one site to cover any filtered routes. This means the choice of /48 is simple geography for us.

We have to then think of the LAN allocation. 65536 possible LANs in a /48.

I think the best advice here is try and be as logical as possible - if you have any existing site numbering scheme, see if that can fit in. If not, try and make one that works with the IPv6 allocations as well.

Hex and decimal

One of the interesting things we see a lot, and do ourselves, is the use of a decimal number from something else, put in to an IPv6 address in some way, but as hex.

e.g. if you had site number 42 (decimal), you may use :42: in the IPv6 Address, even though that is actually 66 (decimal) as the "42" is in hex.

Given that the only reason to use numbering from something else, such as a site number, protocol number, machine number, dialling code, etc, is for your own convenience, then I think this makes sense. You do not want to have to be doing hex/decimal conversions to see and use the "logic" you have created here.

Server addresses

If you have a server for some reason, then you will want it in DNS. You could use an automatic address, but that means change of DNS on change of Ethernet card or move to a new machine, etc. It seems logical to make a server address a fixed address.

What we came up with, and I would be interested in comments on, was using the protocol number in the IPv6 address. E.g a web server on port 80 includes 80 (hex) in the address. E.g. 2001:db8::80

In practice we almost always have multiple servers for anything at a site, usually labeled A, B, C, etc. So we chose to make the protocol and instance in the address.

e.g. the 3rd web server at a site is 2001:db8::80:3

Of course, this begs the questions, if we call that c.webserver then why not 2001:db8::80:C but there are cases where we go over F in lettering servers, and so that breaks. Numbering is fine, and maybe we should have 3.webserver rather than c.webserver anyway? Of course I is :9, J is :10, K is :11, up to Z being :26, even though "10" is 16 (decimal) :-)

Bear in mind that you can give a machine multiple addresses, so you could stick to this protocol based address numbering and allow services to move between machines while keeping addresses. e.g. 2001:db8::443:3 and 2001:db8::80:3 may indeed be the same machine (one for https and one for http). You could even ensure the services bind only to the address specified for them.

Automatic addresses

They work. As long as you don't change the Ethernet interface they are consistent. They can quite validly be what you use and what you put in DNS.

We are moving away from this for any servers (as above), but not sure it is a good or bad idea to use automatic addresses.

PC addresses

PCs on desks can, of course, get automatic addresses. We are not sure if this is the best choice or not. It helps a lot in renumbering but makes it hard to track for reverse DNS, etc.

We did think of matching some other scheme, such as peoples telephone extension numbers being part of the IPv6 address.

Privacy addresses

Arrrg! Who was so paranoid to invent this. I cannot even turn it off on my iPad. I have even set my LAN to use DHCPv6 and flag the router announcements as "Managed", and it still does privacy addressing. I have no hope of having sensible reverse DNS on access logs to our systems.

OK, yes, I can see why some people want this, but I can't see me using it (apart from the iPad).

EUI64

We would love a way to set the bottom 64 bits in a config, but leave the top 64 bits (as well as DNS and router) as automatic. This makes renumbering easy, and DNS would be a simple search replace if we did. But I don't know if that can be done on any PCs.

Names

Of course we all want to be clever. Guess who's site is 2620:0:1cfe:face:b00c::3 though we are surprised someone else is not using 6009:1e in their addresses...

So do we make our machines only use A B C D E F I Z S T O ?

We have managed ec-office (ec0f:f1ce), and faceless (face:1e55), and even boxless (b0c5:1e55).

I don't know if this is good or bad. If the words can be made obvious enough, maybe. What do people think?

IPv4 address based

One thing that we have been told, and I am inclined to agree with, is not to base static IPv6 addresses on IPv4 address. Don't make the machine with 203.0.113.123 use 2001:db8::203.0.113.123

For a start it does not help readability as it shows in hex (e.g 2001:db8::cb00:717b).

Secondly it will get out of step, and then be even less use and could cause more confusion.

Router address

The first address on an IPv6 LAN is special, as it is an anycast for the router. We have used that with no problem for that purpose.

However, it seems linux does treat it as special when accessing it. I am not 100% sure why (someone quote me an RFC). I.e. a machine accessing it should simply ND for it, and get the address. linux seems unwilling to answer a ping from such an address. Maybe something else is odd about it.

What we have tended to do is put a router address on a LAN as ::1, but this being defined as a VRRPv3 address between the routers. I would rather use the zero address to be honest, but if linux is being special about that it is easier not to.

In general you do not have to say the router address as it is set by router announcements advising the MAC to use. If using VRRP you the router announcements should announce the special VRRP MAC address to ensure fast switching. However, if manually configuring a box so that it does not get an automatic address then we are having to tell the box the router address. So we have to consider what address to use.

New FireBrick release

As I said, it has been a slow week, largely down to my not being well. I think I am finally getting over it, but can't be 100% sure. Last week I was fine some of the time, and then would suddenly get a temperature and feel ill...

Anyway, I did manage a few things this week. A couple of minor RADIUS niggles have meant that I have done a FireBrick release this weekend. The new SIP PABX is in the release for the first time with several minor improvements since the beta release I did. It is still experimental, but do let me have any feedback. OSPF is not in there, sorry.

We have a new model, FB6502, which will be the heavy duty SIP gateway product for telco usage. At present it is just starting out with the basic SIP PABX stuff like the 2500/2700 but handing more calls.

In A&A, I have started a rolling LNS switch over to the new code, and that looks fine. I have also updated some of the core routers to the new code, and that looks fine. We do upgrades in stages for obvious reasons, so over the next week or so they will all be upgraded.

I renumbered all the IPv6 in the office - that was not quite seamless, but not far off. Right now I am trying to find a way to configure a linux box (fedora) to have a pre-defined EIU64 and use the router announcements for the prefix. i.e. allow us to "hard code" IPv6 addresses within a LAN but in a way that allows renumbering of the prefix if we need. The DNS then becomes a simple search/replace, but it means the addresses are more "meaningful" which is useful for diagnostic purposes. I am not sure if that is possible. Maybe I need to make a patch :-)

We plan to do renumbering on the Maidenhead LAN, so many machines will get new IPv6 addresses.

I looked at our blip graph for LNS connect/disconnects this morning and it was so low level that it had scaled the graph so the times day were partly obscured by the graph. i.e. it is too quiet... [it is bad luck to be superstitious you know]

So, next week. Two days tied up on a FireBrick training course (still some places for anyone that wants). Meetings on other days. Not sure I'll get a lot of coding done.

So, I am thinking I may take today off. Watch the F1. Play some WoW. See how it goes.

IPv6 tunnels

The FireBrick makes sending IPv6 over IPv4 tunnels very very simple. A route statement is all you need and you can even do it via BGP if feeling adventurous (next hop in 2002::/16).

Because of that we had the office and Maidenhead (MH) tunneled from London (THN) still. This is the case since many years ago when we first set up Maidenhead and the transit provider did not do IPv6. The route announcements on subnets even did 1480 MTU to simplify matters.

It just works so we had not changed anything even though it has been a year or so that the carrier did IPv6.

Anyway, in a burst of activity to clear the white board of one more item, Paul set up IPv6 transit today, and I went through the config moving things off tunnels.

It is interesting. If we announce a /48 from here we see it "on the Internet", and importantly transit in London and HE in London. If we announce any smaller we do see it, but only from the peering with the Maidenhead carrier. So their upstream filters smaller than /48. Not a huge surprise but nice to know. Anyone filtering larger than /48 would route to THN and from there to MH. So not an issue.

Right now we have some sub /48 blocks in MH, but that will change. Customer routing in MH is all on a dedicated /48 so no issue. I suspect we'll move to a separate /48 for MH soon. It means a bit of renumbering, but nothing too drastic.

Anyway, nice to have clean 1500 MTU IPv6 at home at last. Something our customers have had for a long time now. 10th year of IPv6 to broadband lines you know, and all that...

Next stop - new IPv4 /24 and IPv6 /48 PI at home... And why not?

Slow week

Once again OSPF has hit back burner. A small issue with L2TP RADIUS accounting is high on the list, and so is VoIP.

I have been making more and more progress on VoIP despite being anything but apathetic this week (i.e. been ill).

We now have media detection done. Basically, no media for a few seconds clears the call unless it is properly on hold, in which case a re-invite every 60 seconds checks it is still a valid call. Very nice. I need to do early media pass through as well, but that should be easy now.

I have still to do call recording, and all the RADIUS stuff and scaling up to the larger model. Will take a while. Separating the RADIUS client stuff from L2TP is hard work - needs to be a lot more generic than I originally coded it.

The office is using what we have now in anger, and all is going very well. The hunt group logic is working a treat and the call pick-up / steal stuff is great.

I have an IP training course tomorrow and other meetings on Friday, and what with still feeling like shit I suspect I will not make a lot of progress this week. Two day course next week. UKNOF the week after. This will be several slow weeks I expect.

That said, it is much better to be working on the office phone system at 5am (as I have this week) than when people are trying to use it :-)

Most Haunted use Star Trek Tricoder to hunt ghosts

Spot the difference...

On the left is a picture from an old episode of "Most Haunted" on Living TV (which my wife was watching, not me, honest). They claim it is one of the devices they use to find ghosts, etc.

On the right is a screen shot of the Star Trek tricorder iPad app.

Clearly one was based on the other. The one on the left looks like a real thing (i.e. real lights) not the screen of something like that on the iPad.

I wonder what the device is, and if it is real or a toy...

Update: Found it! Most Haunted use a Star Trek Tricorder to find ghosts. Ha!
http://treknostalgia.blogspot.co.uk/2009/04/trek-tech-tricorder-mark-1.html

No wonder they did not show the bottom of the device on screen :-) 

apathy

Well, ill again - all right this morning but by 2pm I was not well - temperature, unwell tummy, headache, and later a neck ache. Arrrg.

Whilst lying in bed this afternoon (yes, afternoon) I pondered the word "apathy". I was contemplating diabetic peripheral neuropathy (which they think I have in my feet) and realising that "pathy" or "pathos" is "ill" or "suffering"... So "at the ends of my body I have a suffering in the nerves due to diabetes" and realised that "apathy" must mean "no illness" like atom means "no cut".

And I failed English at school (like this is English!)

Well etymonline.com says this, so I was right. Apathy is not bad, it is good, it is "not unwell". If only I felt apathetic now!

apathy

c.1600, "freedom from suffering," from Fr. apathie (16c.), from L. apathia, from Gk. apatheia "freedom from suffering, impassability, want of sensation," from apathes "without feeling, without suffering or having suffered," from a- "without" (see a- (3)) + pathos "emotion, feeling, suffering" (see pathos). Originally a positive quality; sense of "indolence of mind, indifference to what should excite" is from c.1733.

Petulant child?

Apparently I am now "acting like a bullying, petulant child" because I have given a customer 30 days notice of termination of service as per terms.

Not sure what I can say to that. He has asked that I "please, go away and grow up".

Well, going away is exactly what we are doing by giving him notice.

He has said "The facts stand and your inability to grasp the complaint, even after this many months, shows that the problem is yours and yours alone." but he has yet to actually make any formal "claim" even though we have suggested he follow our customer complaints code several times now.

He has posted a negative comment on ispreview, which is a shame.

Basically, there is no way we can provide a service that (a) never has a major outage, (b) can force his router, which we did not supply, to reconnect after an outage, or (c) can make a third party mail server not bounce emails he has sent.

So what can I do? - with those expectations I cannot provide service! - so have given him notice and a migration code (which he has not used).

What do you do?

It is funny, people give us notice all the time. We don't accuse them of "acting like a bullying, petulant child", we simply cease or migrate their service as requested with no hard feelings. Why is it that doing the same thing from our side is seen in a different light. Odd...

Oh well.

P.S. Looks like he is finally using the migration code - sorted!

Broken BT promises

I was told saying "our favourite telco" does not help if I am worried about defamation, the main thing is to be factually correct. This is factually correct.

The problem I have today is that BT are ignoring me, yet again.

A customer's line has been off for a month. On Friday BT finally determined that the fault was a problem with the DSLAM port. My understanding is that this type of fault can be diagnosed using a standard automated TAM test (takes a few minutes) and "fixed" by moving to a new port (takes around a day). It has taken a month and is still not fixed.

They have not sent engineers when promised, sent the wrong engineers, cancelled fault reports, cancelled escalations of faults, refused high level escalations, and allowed a fault to continue for a month. Yes, a whole month with no service.

My staff have been chasing several times a day.

Our new BT account manager had managed to make a lot more progress over the last few days, getting the Director's Service Office involved, which is how we got as far as actually diagnosing it was a DSLAM port issue.

Even so, the latest is, now that they have identified the fault, they have ceased the line and are re-providing it. The promise yesterday was "I will have it back on tomorrow morning".

This morning at 07:15 I chased to find a more exact time, and they said they were not sure want time.

Morning came and went, and no reply to email at 09:01, 12:01, 13:11, 14:07, 15:03. I have emailed again at 17:11. echat to HLE says no representative, so email was only option.

Whilst today may be "working hours" by BT's definition, it is not for me or my staff.

Once again BT waste my time, damage our reputation as an ISP, and mess our end user around.

Recently, I asked if I should fall out with BE, our other supplier, and we are having a meeting soon with them to make sure we don't fall out and we all know where we stand. I am sure that will all be resolved and both sides are keen to understand why we had a dispute and what we need to change to fix that.

But what do you do with BT when you have little choice but to deal with them.

There is no ADR I can use to chase BT. Shame, as so many shorfalls. If I am lucky we will get a few pounds compensation, but as they kept sending the fault back to us (unfixed) we probably do not get that. They have the cheek to charge us for this service for the month as well. We refund our customer, and that is our loss as is all the time spent on this. I think our customer has bigger losses, sadly.

I am appalled. I only hope we get this customer back on line soon. I do feel sorry for them.

The sad thing is that I remember when we started broadband and things were good. BT were good to deal with most of the time. Their fault desk gave the impression that they actually wanted to fix faults. They did not try to micro bill for everything and they did not charge to send an engineer. Why has it all gone so down hill? Why are the majority of interactions with BT a battle. We should be "on the same side" trying to help an end user and fix a fault.

I wonder if there is a way to get BT back on track. To actually want to fix faults not fight ISPs. To actually want to provide a quality service. We want to fix things, and we are happy to try and work with BT to do that, but are they? I don't think my 5p BT Group plc share gets me much say from their side.

To give you an idea of what we have to put up with - this is just what they are doing this weekend on this one fault after they had promised the line would be back on Saturday morning:-

• Sat 07:41: Got a reply to my email of 07:15 but they could not confirm time when line will be working
• Sat 17:00: No reply to emails of 09:01, 12:01, 13:11, 14:07, or 15:03.
• Sat 17:24: Since starting typing this post: one reply to my 17:11 email and one to my 15:03 email. Promised update at 6pm by Karen
• Sat 18:00: No update and 6pm has passed. What a surprise.
• Sat 18:01: Latest update says they have not in fact even expedited this as waiting on the order getting to "committed" stage. WTF. This was promised for "this morning".
• No reply to emails of 18:03, 18:09, 18:25, 18:48, 18:53, 19:00, 19:07
• Sat 19:35: Latest news is that it seems BT plc have no control over when BT plc's systems will show the order to be committed state. I would expect BT plc to actually have control over BT plc's systems myself.
• Sun 06:41: Finally an update, and they have cancelled the new order for the line replacement and can't do anything until some planned systems work is done. So yet more excuses and more problems.
• Sun 12:04: An hour since BT's planned work was finished and BTs systems restored. We were told "The very moment that they are I will do so, and keep you informed of the progress." but that is another broken promise I see as they have not even an hour later in spite of emails at 11:08 chasing. No reply.
• Sun 13:06: They are finally placing the order. 
• Sun 13:53: They claim to have placed the order. Now to wait for their systems to think about it for a while. I do not have high hopes of this being fixed on a Sunday somehow.
• Sun 15:31: Order still not showing on BBCR (i.e. BTs systems not updating us) and no reply to email of 14:54 asking for new circuit ID. Not convinced this is going to happen.
• Sun 17:00: Still waiting for order to show on BBCR. Still waiting for confirmation from BT of order state and circuit number. Getting silly. 
• Sun 18:00: Still waiting for order to show on BBCR but BT have confirmed the circuit ID. It does not normally take this long for an order to commit. I have my concerned that this will be another "system problem"
• Sun 19:19: BT say they have expedited the order - now to see when it will be jumpered.
• Sun 20:02: Finally the order shows on BBCR - as issued, not committed. Hmmm
• Sun 21:01:  End user logs in. Hmmm, odd as BT seemed to think they would not be able to do anything until 07:30 tomorrow. Can't do line tests or anything, but may in fact be working. Yay!

I would say that was a successful, if rather long, weekend. Thanks to Stuart and Shaun for their input on this as well.

Now we need to get a proper report from BT on why this all went so wrong and what they (or we) can do to improve things in future.

It is tempting, as some have suggested, to publish the bad experiences on fault reports like this from time to time, but I would want to ensure we publish the few good experiences we have as well. Just occasionally we get a shock when someone actually fixes a fault promptly and efficiently. I'll post if I see such a case in the near future.

Back to work

Actually managed a whole day off - that is quite rare - and I am pleased that the office and customers did not pester me all day - thanks. I watched all there Sky Terry Pratchet films and eat paracetamol.

So far today, feeling quite a bit better, so designing the call billing system for the new VoIP server, and trying to find excuses to put of OSPF a bit longer.

Also chasing a fault which has so far taken our friends 30 days to get fixed, and it seems it was just a faulty DSLAM port all along. They did not even need access to end user premises and could have found the fault using an automated TAM test in the first place. They could have fixed it within a day by a "lift and shift". How can any telco take 30 days to fix a simple fault. I should bill them for the dozens of hours me and my staff have spent on this, but I don't think the contract allows for that. It should not need to get the Director's Service Office involved in a simple port fault. Shame. I do feel rather sorry for our customer on this one. That said, I think they may consider the "two lines is better than one" logic we normally try and promote for any system critical applications...

Update: Guess what, when they say "I will have it back on tomorrow morning" is the line back yet? Its 12:00. No it is not. At least they are consistent in that they they have broken yet another promise. Arrrrrrrrrg!

Anyway, catching up to be done...

Happy Birthday Pauline :-)

Off sick!

Well, I have decided to take the day off sick - rotten headache in spite of paracetamol. Somewhat rare for me to have time off.

OSPF and VoIP can wait a bit.

Targets for customer services

A&A have high standards internally, and aim to provide the best technical support we can (as a very technical ISP) and obviously provide good customer services.

This target is a tad vague and not one that is easy to measure. The best measure of success is the glowing reports we get on thinkbroadband and ispreview.

The fact we have a couple of less than glowing reports on ispreview has sparked much discussion in the office to try and work out if we did anything wrong. I could go in to details, but someone not reading invoices and not knowing the VAT rate changed is difficult to address, as is someone that has an objective to take us to ADR "to waste our time" and not actually trying to resolve a dispute. Even so, we want to work out if we could have handled these cases better.

It is tricky to have objective targets for "customer service". You have to always be careful what you measure as you end up with a machine to make good measurements and not actually do what you want.

I have generally been happy to work on this vague target and look carefully at feedback rather than trying to make specific measurements.

Of course, we have a separate target we work to in terms of "did we do something wrong" which is the agreed contract terms. If we met the agreed terms we have done what we agreed - everything above doing that is an internal target we have.

But should ISPs and telcos be required to work to some higher standard - some customer service target that we are expected to meet, and penalised if we do not meet it?

I assumed not. I assumed it would be down to ISPs to do business in an open market and those that have poor customer service (whilst still doing what they agreed) would simple have less business as people are prepared to pay for good customer service.

Yesterday I went over all of the ADR stuff yet again, providing details for our MP, who is interested in pursuing the matter. I realised that the whole ADR case only makes any sense if there is some "higher standard" of customer services that we are, as an industry, expected to meet. After all, they agreed we were not in breach of contract and all the "shortfalls" were failures to meet some unspecified and unknown targets that they have invented.

So we'll try again and ask them what these standards are exactly?
How do we measure them and so ensure we meet them?
What is the prescribed penalty for failing to meet them in various ways?

Only if we know this can we sensibly work in this business environment. We can't have unknown standards with unknown penalties.

Lets see what they say...

In the mean time, we'll stick to our aim of being the best we can.

Polycom works

Well, we have tried a few different phones now, and finally found an old polycom model. It has taken quite a while to configure it as every change needs a full reboot, but it works.

Even managed a call transfer on it. Audio works. DTMF works. Hold works. What can I say!

So far that is snom, cisco, gigaset, gigaset DECT, gigaset IPv6, linksys, polycom, and grandstream.

Today we are working on testing carriers, and have it registering with asterisk now.

What next

Well, I have two big jobs to do on the FireBrick.

1. OSPF as a terminal node injecting routes for L2TP and the like.

2. More work on VoIP such as RADIUS accounting and making a big core network telephony switch.

The next step on VoIP is relatively small - RADIUS accounting for the calls and a linux/C app to collect the stats and put in to a mysql database. So tempting.

The VoIP then moves on to be more scalable and having RADIUS authentication and call routing - well loosely based on RADIUS, and the linux/C apps to do that.

I also have all the NAT testing to do. (don't forget kids, NAT is evil, so don't try this at home).

But I promised some people OSPF, so I have to be strong and try that first. Hmmm.

Where is an ADR case to distract you when you need it?

Let the fun begin

OK, I have issued a new beta release of FB2500 and FB2700 code with the initial (experimental) VoIP code in it.

Details on http://www.firebrick.co.uk/fb2700/voip.php

Have fun and let me have any feedback...

P.S. couple of alpha's since - let me know if you need to be able to load alpha's.

DTMF

SIP phones have a number of ways to handle DTMF (i.e. tone dialling).

DTMF was designed for audio connections really, and so does not compress that well. Thankfully one can use a-law everywhere these days so DTMF is fine in-band - i.e. just as audio. So that is the simplest way to handle it, in theory.

However, as SIP can use all sorts of compressed codecs there are different ways to do it - a common one being to use "telephone-events". This is a different coding for the stream of audio data to say "this is a DTMF digit" but still sent in the RTP stream in place of the audio.

The problem is when you bridge something that does understand telephone-events to something that does not.

The answer, which I coded this morning on the FireBrick, is to take each 20ms DTMF "message" and turn it in to 20ms of actual tone. As they say, simples!

Well, it works! I am quite pleased. I can bridge a device like a SNOM sending DTMF as telephone-events to a carrier that does not handle them and call an annoying call gate system and press keys and it works. Yay!

So, we tried with a gigaset DECT system. Does not work. Arrrrg. I have a packet dump and I can see why.

It sends the audio (i.e. from the microphone) at the same time as sending the DTMF signalling!

What is worse is the DTMF is sent 20ms in the past. If the DTMF was sent first and then audio with the same timestamp, I could simply discard duplicates and that would work. But no, there is audio, and then DTMF time stamped 20ms before the audio just sent. Then there is audio and then DTMF again, and so on.

Given that the gigaset can, instead, do INFO messages as a means to send key presses, and seems to default to it, I may just make that work rather than trying to bodge something for DTMF on the RTP stream.

Oh what fun.

Seeing reason

Just to clarify, when I say BE, I mean the company we buy the BE circuits through, not BE or O2. I don't want any confusion here or defamation claims :-)

However, the supplier has seen reason in this case, thank you.

Now we want to make sure that the processes and contract terms are more clearly agreed to avoid any issues in the future - so we aim to work with them on that.

I don't like being confrontational, but I do have some principles. Unfortunately principles clash with doing business on occasion, and that can cause issues.

So do not worry - we are still doing BE lines, and we are working with the suppliers to ensure we are working to the same principles.

Thanks again.

P.S. thanks for all the comments on this issue.

...More...

Some comments I have had on this are, quite rightly, accusing me of throwing my toys out of the pram. I understand that, and yes, it was a bit like that, sorry. This was the culmination of weeks are arguing over this. In practice it would take a lot to stop us selling BE lines, and the contingency for this is something I have considered. The most likely being that we work with another ISP who does the BE lines to provide even more redundancy where people need it. I am always keen to ensure that a dispute like this would not cause our customer's any major problem.

Do I fall out with BE?

Well, a tricky one.

They want me to pay for an engineer that went out to fix a line when the cause of the problem was a third party.

I refuse to...

It would be a shame to fall out with them - we would need to sort hundreds of lines that are with them.

Maybe we need to talk to some other LLU providers - perhaps Virgin or TalkTalk.

Lets hope they see sense.

P.S. Of course the answer may be we pay but never sell another line with them ever... That is their choice.

P.P.S. This is a case where the fault was not caused by customer equipment. A case where even BT would not charge us. An interference issue from a third party.

How hard can it be?

The Online Safety Bill says that ISPs have to filter pornographic content unless you say you want porn, and are over 18. It is not law yet.

I have been trying to come up with an analogy for how stupid this is.

The proponents think it is easy, and no different to controlling the content on television, or so it seems.

The best analogy I can come up with is that you are demanding people that sell glasses have to filter pornography being seen through them.

I think people realise that it would be stupid to say that, but this seems to be what the bill is, in effect, saying. ISPs provide access to the Internet. They are not providing the "content", just access to it. Asking the content providers to have age verification and filters would be possible, except they are not all in the UK. Asking the people that provide access to the Internet to block it is crazy.

ISPs have no way to tell what is and is not pornographic at the packet level.

Systems to block access to all pornographic images would be technicially impossible. Even the existing IWF filtering systems don't actually aim to block access, just stop people accidentally coming across certain web pages. But this bill calls for actually blocking all access.

I could email you and encrypted file that is a pornographic image, and there is no way the ISP or mail servers can tell. I only have to find one way around it to prove it is impossible, but there are many ways it will not work.

As an example, the ban on newzbin by a court order is not actually stopping access as it just blocks http not https, or indeed any number of other ways to access the site. The ban has, instead, created huge publicity for the site.

Porn sites will be no different. Yes, a few larger sites can be black-holed at an IP level, but that is not a total block or even close. Anyone with access to google will be able to search for ways around any blocks put in place. It is a waste of time. I suspect teenage boys will have access to google and some motivation to find porn.

It is however a huge costs for ISPs to do this.

It is also making people specifically tell their ISP that they want to access pornography. The ISP then has that sensitive personal information to handle and not leak. The ISP also gets to track all the attempts to access pornography now they have a system in place to block it or allow it based on subscriber preference.

Of course, the block is network level so affects a whole household. Even if the block was 100% effective, if the parents want access to pornography then the block will not stop the kids accessing it via the same Internet connection. So again, pointless.

A&A have asked people to confirm they want unfiltered Internet access as part of the sign up for some time. We have not (until now) asked people if they are over 18. Our service is unfiltered, and buying it is opting out of any filters. Sadly this means only those 18 and over will be able to buy our service if this law comes in to force.

Also, a subtle point in the bill, is that it requires all computing devices to have filters. This is filters generally, not just for pornographic. This is any device that connects to the Internet and can download content. So my TV, and my SIP phone, and so on. They all have means to download content. Indeed, most things these days have means to download content even if that is just to update their own software and they are not in any way a computer. This is a far reaching bill!

Who knew? SPAM may have uses

Most email is SPAM. You can tell the SPAM by looking at the content in many cases, but as well all know that is not 100% reliable.

However, if you are grabbing only the communications data, i.e. the "from", "to" and date/time, you cannot tell the SPAM.

So most of the data you collect is bogus. The email addresses are mostly not even real, as sender or recipient, and the apparent associations between people are not real either.

It is hard to see how this data can be useful even just for investigating connections between people, and clear that it cannot be trusted as evidence of anything.

Of course, if you then get some sort of packet snooping systems in place you find that you can be collecting a lot more data.

It would be trivial for someone to make an application that connected (apparently) to a mail server and apparently tried sending emails from lots of email addresses to lots of destinations. The email would not be real. The mail server would not be real. The data would just be the headers to catch the snooping systems.

This would be low levels of upload on a broadband line, constantly, 24 hours a day. It could even stop when you are using the Internet so as not to disrupt real usage.

It would make the data collected even less useful. It would simply poison the database that GCHQ are collecting.

So, who knew, SPAM may have some use after all!

April fool 1984?

A number of people thought that the BBC article Email and web use 'to be monitored' under new laws had to be an April fools joke.

After all, even the government are not crazy enough to think they can legitimately spy on everyone's email and text and tweet, speculatively...

But no. Apparently it is real. There was a talk on this at ORGCon2012 but few details had been released by the government of what they planned.

It is still not entirely clear and I am sure that there will be a lot more detail in time. Obviously civil liberties groups are up in arms over this, with good reason.

The main issue here is the issue of "communications data". For a long time the authorities have been able to get details of who called what numbers from telcos. The problem is that this is not so clear now. Are your facebook friends list just communications data? Where is the line drawn exactly.

This also goes way beyond what was done before - which was simply expecting the incumbent telco to search its logs which it had collected for billing purposes. Now we have a situation where there may be no logs (not under UK jurisdiction anyway). People use hotmail and gmail and so on - and not their ISPs mail servers, so there is no record of who emailed who for an ISP to search. Even where a telco is all UK based and their servers are used, they may not actually collect communications data. After all, if they have "unlimited" package they don't need to. Up until now there has been no requirement to collect extra data for law enforcement, and indeed, under Data Protection laws, there is reason not to collect any data you don't need. Collecting extra data, and, importantly, keeping that data safe and secure, is extra cost for ISPs and telcos.

The proposals seem to suggest that they want monitoring at the packet level to track who emails who even if using some gmail web page to do it, or messaging on facebook or twitter.

This is crazy, from a technical point of view. Anyone that has ever tried to screen scrape a popular web site will know it needs constant tweaking. You can't just put a black box in and expect it to work - it has to handle every new application that comes along that allows messaging and every change that is made by the web designers and application designers, none of which have to publish any spec or notify the UK authorities of changes. So to do this you need not only hugely powerful monitoring boxes, but boxes that allow remote administration and update - so could easily start monitoring lots of other "stuff" with no visibility of the ISP or their customers. Thin edge of the wedge?

Of course it is also totally impossible to win this - anyone that has any reason to hide their communications data can do so - it is very simple.

What makes things even worse is that it is not just the "bad people" that can easily hide their data - it is happening as a matter of course. Web mail applications are using https - encrypted from the users device to the server that may not be in the UK. The servers and user computers have more than enough computing power now so that strong encryption is the norm.

Of course, I was struck by how silly this is today when I downloaded wordfeud on my iPad because my son's girlfriend's parents play it (long story). Suddenly that is new communications data - it has in-game chat.

AAISP have no intention of installing any montioring equipment. Sadly, if the government have any sense, they won't expect us to - they will install it in large carriers or at the borders to the country - like China.

As an ISP, we already explain to customers about running your own mail server and using encrypted mail transport and end to end encrypted emails. I can see us explaining things like Tor in more detail soon as well. After all, if criminals can hide who they are communicating with, surely law abiding citizens should have the same right?

Anyway, there will be a lot more on this over the coming months I am sure. Lets hope that groups like ORG can fight this effectively.

P.S. Nice diagram, thanks to Alec from ORG:-