If I am on a course

Well, a busy two days running FireBrick courses, and I have to say that Jon gave me a run for my (his) money.

Probably the first time I have had a "me" on one of my courses! I must be a nightmare to train me!

He was always one step ahead of everyone else, and at least two slides ahead of the presentation, whilst always checking his emails.

That said, I like this - made the course a lot more fun, and a hell of a lot of suggestions and even bugs/typos found as a result, which is excellent.

It was a pleasure to do a course for someone that knew his stuff, and was there to learn.

Thanks Jon.

No such thing as 7 hours fix

Our favourite telco offer a service that is a 7 hour fix for broadband. It costs extra money, as you would expect. We have not yet used that service.

Now, for some time we have said that any fault, even if on the normal 40 hour fix, or 20 hour enhanced care level, if they fail to meet that, should be treated as a 7 hour fix expensive fault. After all they are already in breach of the agreement and they have the resources (the engineers for the 7 hour fix) to fix things...

Tonight I had an interesting echat - a simple matter of a faulty line still not fixed. The engineer was booked for today but no-show. Engineer lied saying he contacted the end user to confirm it would not be tomorrow... Lying for any sort of gain being a criminal offence under the Fraud Act. They committed a crime, as I see it. They knowing lied for their own gain. The police should be involved.

Now, on an extensive echat with our favourite telco, Alok confirmed that there was absolutely no way an engineer could come out to fix this in the next 7 hours. No way at all...

Adrian Kennard (19:07:17):
None,. so the 7 hour fix service you offer is a lie, yes?
Adrian Kennard (19:07:25):
You could not meet that under any circumstances?

Alok xxxxx (19:09:34):
I am afraid so, at this time we cannot deploy an engineer, Adrian.

What can I say - that means the service for 7 hour fix is a lie, a fraud, criminal.

I await their explanation on this matter.

And (as I said in the echat), good luck to Alok in his next job.

Just to clarify - they offer a 40 hour fix service. They offer a 7 hour fix service. But surely, 33 hours in to the 40 hour fix service it is then a 7 hour fix service? That is the very definition, is it not?

Favourite telco

Has anyone else looked at today's Dilbert and wondered if he is talking to his favourite telco?

Busy week

Finally - end of the week - get a Chinese and relax in front of TV maybe...

UKNOF was fun this week. Always nice to mingle with geeks, and was not a bad venue so thanks to BT for that. I love the chatting to ex demonites at the social - getting rare to find people that have been there and done that with every conceivable type of networking quirk you can imagine. Oh, and as per everyone that asked me at the social... Them:"We saw your blog"... Me:"Mark all packets as LCP"... Them:"Damn, was hoping it was something I could do on my CISCO"... What can I say? "Should have gone to FireBrick". Thanks BT for letting us get an edge over the competition...

We had a quirk worthy of that crowd this week... As you may know half of BT's 20CN network was broken when using native IPv6. We got the whole "IPv6 is not supported" story on that, lovely. It meant we had to pad PPP packets under 74 bytes to make it work. Mad. FireBricks FTW though, as ever. BT have fixed it (well done BT). But we still had it turned on for 20CN lines on our end.

So we get someone saying they cannot get email, but, and get this, only... (a) IPv6, (b) Technicolor router, and (c) windows. Even loading MacOS on the same hardware was fine, or changing the router to a Billion, worked, and IPv4 was fine!!!!

We turned off IPv6 sub 74 byte packet padding (as BT have fixed it) and it works. Looking at logs, both small and large packets fine, but some in the middle were "lost" somehow. My guess is padding 74 byte PPP created padded Ethernet which is fine under 64 bytes but when over must break something in windows. We have to do some careful testing to find the cause - may be technicolor and may be windows - who knows. I don't know what the rule (RFC) is on PPP to Ethernet if padding should be removed. I bet it is some checksum offloading somwhere. All I know is that (as per RFC) padding on PPP is always valid so not us :-) It is solved by this tweak, but I would love to know the cause. We have to test this in the lab...

The ex-demonites at UKNOF (sorry guys I am shite with names) had similarly bizarre stories from times past and that was fun.

Of course, I missed last train, but even a £145 taxi back home was cheaper than a premier inn in London and a train back next morning, and was my own bed even if it was past 1am.

And the good news - seems Tom's parents have him named on their insurance and he may be able to claim for his camera kit - well done and best of luck.

And then world IPv6 launch - yay! But that really is nothing to us - we are in the tenth year doing this shit, and we have been allocating /48's to all new customers for over a year with free IPv6 pre-configured router since before last year's World IPv6 Day event. My grand plan is a special for existing customers to get a cheap IPv6 router from us and move beyond our current 25% of lines with IPv6. Would be cool to push that to over 50% by June 6th... Lets see.

Have a good weekend all...

Bad luck to be superstitious

Well, it is, in that if you are not superstitious then the crap that happens is just crap that happens and not bad luck as such...

So, Friday 13th, and clearly the network switches in THN know they are about to be replaced next week as one of them starts going "iffy". Well that is the best explanation we have at present for flaky traffic between two boxes on the same switch this morning. As ever, a total failure would have everything falling back in seconds to the secondary systems, but "iffy" is the system engineers worst nightmare.

Of course it happens when we have a completely separate minor issue affecting incoming connection RADIUS on 20CN lines due to our favourite telco jumping the gun on a config change. Well, an issue that would be minor. Basically, it only affected new connections, a couple of lines, and would take us minutes to solve. It took longer because all hell broke loose when the switch went iffy, and as we dropped all sessions it meant nobody on 20CN could reconnect! Anyway, all sorted now.

The good news is the new rack is being kitted out - we have several gigabit fibres in already and a load more links to set up and move over, new switches, new routers and LNSs (FireBrick, of course). All moving to multiple gigabit operations hopefully by the end of the month.

Of course that is not the only "crap that happens", but that is a different story.

Happy Friday 13th.

80Mb/s FTTC

Well, it is impressive what you can do on a short bit of copper :-)

We have a number of customers on the FTTC 80Mb/s trial now. One has 79.7Mb/s and another has 57Mb/s. This is pretty impressive. The uplink is around 20Mb/s as well...

Seems the trial is not going badly - the lines are syncing up at nice high rates, and the rates are correctly coming through to us.

Seems a few minor teething troubles getting the profiles right in the middle somewhere - but this is a technical trial, so not at all surprising.

Fun...

Actual speed test on 58M sync line

RIPE PI

They have stopped talking to us now, which is a shame.

The problem is quite well summed up in the RIPE policy document on this. Basically, until recently, PI space was issued without any contract.

The problem is that, without a contract, there is no way to ensure the end user follows RIPE policies, as there is no way to ensure PI space is returned if the end user does not follow RIPE policies. This is indeed a problem.

The solution that the community has come up with is to, err, make a policy which requires end users to have a contract.

So the solution to not being able to enforce policies is make a new policy, and to pretend that this new policy is enforceable.

Once the new policy is followed (contract in place) then all policies are enforceable.

It is a paradox though. If the new policy is enforceable, then so are the rest, by whatever means the new policy is. That means a contract is not needed because RIPE do have a way to enforce RIPE policies, so the new policy is not needed! If RIPE don't have a way to enforce policies, then the policy to have a contract is not enforceable so pointless.

I am appalled at the way we as a community have tried to do this. I should have been more involved in the discussions on this as a RIPE member myself. I was not. I do agree that all new PI space should have a contract - that is fine.

The key thing here is the wording in the policy about the end users returning resources. If an end user does not follow this new policy but does not return the resources then they still have them. If RIPE could simply take back resources then there would be no need for a contract with existing PI holders, but the policy document clearly recognises that end users would have to actually return the resources.

If the end user still has them then RIPE NCC have an obligation to ensure the RIPE NCC database is correct for use by all its members. If the end user still has the PI space then RIPE NCC should correctly record that in the database - otherwise the database is useless.

So far we have a string of unanswered questions for RIPE NCC on this...

1. The contract is being forced by coercion or duress in that RIPE NCC are saying they will remove database records which will effectively break the Internet connectivity. Forcing someone in to a contract is against the principle of contracts and a contract formed under duress is not valid.

2. The contract does not offer the end user anything. It gives the PI space (which the end user already has) and applies restrictions and gives third party rights, but does not give the end user anything new. It does not guarantee a RIPE database record and explicitly does not guarantee Internet connectivity which is what the end user wants. A contract with no consideration is not a valid contract, which means that the third party rights to RIPE NCC are not valid either.

3. If the end user does not enter into this contract then they are in breach of RIPE policy. But there is no contract requiring them to adhere to RIPE policy so that does not matter. They do not have to return the PI space to RIPE. If the end user still has PI space then transit providers should honour it (even in the absence of RIPE database entry) and refusing to do it starts to raise issues that relate to the Communications Act and could involve OFCOM. Forcing people to accept IP announcements outside of the RIPE database is bad - the right thing to do is for RIPE NCC to ensure the database is correct and reflects that the end user has not returned the PI space.

Current plan is end user signs a contract, though has a cover note explaining that the contract is signed on the basis that the end user knows it is not valid or enforceable and that the end user has no intention of granting rights to RIPE NCC as a third party. If it can be shown the parties to the contract did not intend to grant rights to a third party then the contract cannot do so even if it is deemed otherwise to be valid.

I think, as a community, we should do better in future.

I think RIPE NCC should not be ignoring me now that we finally have a fairly good and well formed argument why what they are doing is wrong.

Bypassing congestion on a carrier network

We have seen much debate on net neutrality of late, and this even relates to the ways in which various ISPs prioritise various types of traffic.

One would hope that this does not apply to a wholesale carrier network. Ideally they charge for usage (as our favourite telco does) and so they always have enough revenue to run an un-congested network. There will be exceptions, but in general the network should cope with no problems. This means that there should be no need to have any priority for certain traffic.

You certainly would not expect a major telco wholesale carrier network for broadband lines to have (a) any links that are regularly congested, and (b) priority of certain traffic types over others.

Well, with the new FireBricks which do PPPoE at the customer end we now have control over both ends of the PPP link on a broadband line. We can change the traffic type of the packets that are carried. To our surprise the result is a massive improvement in performance on some lines that have congested back-haul links.

We are not alone in noticing something odd. Another ISP has noticed this as well and they, and we, mentioned it at an ISP forum working group last year. The telco denied there was anything going on.

So we set about testing this. It has been hard work - we needed the PPPoE code in the new FB2500 and FB2700 FireBricks; We had to find a really good example line that was congested every single evening; We had to set up testing to confirm the level of congestion clearly; Finally, we had to make code to allow use of different PPP packet types and test the results.

The result is a unique combination allowing us to give A&A customers that have FB2500 and FB2700 an advantage at no extra cost for the service. Please don't rush out and spend money on a FireBrick if you do not need one - they are not cheap and usually only make sense for small business customers and not home users. But if you do have congestion, especially on a 20CN line, and the premium option on the line is not helping, it is another step we can try if you are prepared to invest in a FireBrick. The main example we have tried this on was bonding three lines where one was heavily congested in the back-haul - this fix helped massively.

We cannot suggest anything underhand is going on here - it could just be a consequence of the way the equipment is designed and set up to handle a congested link.

The real fix is for the telco to stop running congested links and believe us when we tell them there are problems. They have clearly stated they will happily put 400 ADSL customers (up to 7.15Mb/s each) on a 10Mb/s back-haul. That is crazy. Thankfully most of the links are OK, but there are a few that are a problem, or were until now...

A&A support staff know to watch out for this type of congestion and we will be adding more tools that end users can access over the next few months which will highlight congested links more clearly.

Of course A&A are no longer alone in using the FireBrick FB6202 LNS. The FireBrick team will be happy to explain the details to any of the other ISPs that use FireBricks. Any ISPs wanting to try them - contact FireBrick or me.

Guess who?

Make it a requirement of the contract that all disputes are raised within 14 days of invoice date.

Make it a requirement of the contract that payment has to be made within 28 days of invoice date.

Don't actually issue and send the invoice until 36 days after the invoice tax point date.

Now what company would ever do that?

Arrrrrrrrrrrrrrrrg!

(FYI I have now said we interpret "invoice date" as the "issue date" which HMRC require them to put on an invoice that is not issued on the tax point, and they can lump it).

Moral compass

Where do you stand on the subject of lying?

This is really one for discussion rather than just a blog.

I find that I have a clear idea on what is right and wrong in this area, and I know some people have slight variations on this one way or the other. i think I am saying what I do correctly, but obviously it is selective memory, self justified and confirmation biased, so I can't be sure either way.

Personally, I don't like lying - however, for me, saying something that is actually true but just has some omissions is not actually a lie as such. Some people will consider that unacceptable. Others will be happy to say things that are not quite true. Some people are happy to blatantly lie for their own gain.

Outright stating something that is not true really winds me up and I won't do it. Well, I try not to - it hits my conscience if I do something as simple as telling a kid there is a Tooth Fairy. Of course this is an area some people are more than happy with if it is a white lie with no real consequences and not for some sort of gain in any way. I understand the logic but I would rather say nothing and leave someone else to tell a white lie. Not correcting them is about as far as I will usually go.

Then there is deliberately constructing something that is technically true, but you know damn well that someone will mis-read what you are saying. This is an area I find very close to the line and not that keen on - but just the "OK" side of it. My tolerance of this is very much based on marketing and adverts - where they appear to be allowed to make technically true statements and get away with it. However, my cynical side assumes this to be the case and I always look for what has not been said but could, or should, have been said. Someone says "up to 100% more shine" or something and I immediately hear "anything from 0% to 100%" - I think about what it actually means if what is said is just about technically correct and assume the worse. I suspect that makes me a cynic.

In general I think that in most cases leaving out details that could have some negative impact is probably still just the right side of "OK" - even in business contracts. In business you expect someone to ask the questions, and state what they want. Consumers, on the other hand, have to be hand-held a lot more and you have to say things that seem obvious, even if they are negative, in order to avoid any misunderstandings later. As a result we have a lot of things on our web site spelling out the limitations on some services as well as the benefits. This is "doing the right thing".

Simply leaving out details because they do not matter is clearly "OK", IMHO. You have to be slightly autistic I suspect to worry about not including every single detail and rigorously tell the truth and the whole truth all the time. This is where you get in to the area of  "does this make me look fat?" and the answer "yes, hideous!", which usually loses friends and alienates people...

Of course there are some grey areas where people think what they are saying is true. If you really believe it, then that is not really lying - I think generally you have to know you are saying something wrong to be lying. Somewhere in between you get the situation with people saying what they do not know is true or not, but could be, so they will say it anyway because they don't know it is wrong for sure (i.e. bullshit). This is the area where I start accusing our favourite telco of lying though. I would rather people said they do not know than make stuff up.

So, am I odd?
(OK, I know I am odd, but in this particular matter?)

How do bankers do it?

I think I have been a bit of a meanie today - I charged someone an admin fee (£5) for sending them their own money back. I feel a bit dirty - I mean - charging money to move money - I must be practically a banker now (eeek!). I talked about turning to the dark side with a windoze box, but banking is going a bit far...

It is, I think, the first time we have ever implemented this even though it is in our terms.

Just occasionally someone will have a standing order wrong by a bit or make a mistake and send us too much or even send a payment twice. More often than not it is simply a matter of paying the correct balance next month to get it right. Rarely someone asks for the overpayment to be paid back. We have even had someone ask for an overpayment to be sent back the day before they get their next monthly invoice which amounted to exactly what they had over paid - they were asked to wait a day and bingo the balance was zero...

It is hassle! As I am sure you can imagine sending people money needs some authorisation - in fact the system won't let anyone but me put an outgoing BACS payment through. So it involves accounts staff and a director. That has to be worth £5?

On one occasion, someone asked nicely and even pleaded, so I had the extra hassle of doing a fast payment rather than a normal two day BACS. Even so we get a handful of these cases a year. Asking nicely helps - anyone who starts making demands and threats is likely to get a cheque by second class post...

But this particular customer has over paid four times now and asked for the balance back. Once might be a mistake, but four times is just careless, and, IMHO starting to take the piss.

Anyway, I can't see my making a habit of it, not like a real banker :-)

Passwords, memory and xkcd

I have various passwords on various systems, as you do...

For our main internal systems I regularly change my password and have, until recently, used mkpasswd to create a password with letters, digits, symbols and so on. Obviously I don't use this password on any external systems.

I would make a point of manually changing password on various systems so I had to type it many times - even so it would take me a few days to remember it, and so risk shredding the post-it!

Recently I changed to using an XKCD 936 password. This is four random words (adjective/noun/adjective/noun). This is long, but has lots of entropy so is a good password. It is important they are random words and not four words you pick yourself in order for this to be a good password.

The first thing I noticed is that I remember it - no need for a post-it note at all. That is to be expected and exactly what Randall was saying in the cartoon. I was able to create a contrived mental image to remember it, just like correct horse battery staple (no, that is not my password).

However, what is interesting is that I still had to think about typing it after days or even weeks. I.e. I knew the password but my fingers didn't. I am now just typing it without thinking, at last, but that took a lot longer than with the old shorter passwords. I suspect it is simply a matter of the length. Not really a problem but an interesting observation.

Another quirk I have noticed. With the old passwords I would immediately forget my old password when I made a new one. This was such an issue I would have to write down my old password just in case I had not updated something and needed to know it later. I cannot recall any of my old passwords from that system. But what is odd is that the new XKCD 936 password is not replacing the old password in my memory. I still remember the last mkpasswd based password and my new XKCD 936 based password at the same time. They obviously use different parts of the memory somehow.

The old passwords appeared to be remembered in the way it is typed, so much so that to say my password (which you never do) I would have the think about typing it and realise what keys I would be pressing.

I will change to another new XKCD 936 password at some point, and I wonder if I will forget the previous one or not. That will be an interesting test.

We will probably be moving to a system of OTPs for many internal systems in future - with keyring code generators. Shame, as I am starting to like the XKCD 936 passwords.

Isn't it funny how the brain works some times.

Happy New Year

I would be a tad remiss of me not to wish all my readers a happy new year.

It should be an interesting year though.

We are pushing the FireBrick beyond the gigabit with multiple gigabit LNSs shared traffic policing in to our favorite telco over multiple gigabit links. This is part of a major network upgrade for us and I am actually quite pleased that this project is being organised by my staff - they are actually doing more "management" now which is saving me a lot of hassle. I have had to do the directorial steering of policies and the like but the work is being done by them. That is cool. That said I am the one that has spent the last week coding the new FireBrick shaper sharing system. We are closing a rack in THN soon that I have actually never even seen - that itself is a testament to my staff.

But there are a lot of things happening. I will remind someone that he promised me IPv6 on data SIMs "by the end of the year" in 2011 too. We have a work around, as ever.

I still think the copyright holders and legislation issues are going to have to come to a head in a big way somehow - maybe this year. It is getting very very silly now. A&A explicitly sell an unfiltered Internet connection which will make it impossible for a court order to require filtering as it would not be the service we sold. It would be an order to break every one of thousands of contracts. That will be fun if it ever comes to a court order. You can bet I will fight that one. And if I lose you can bet the some new company pops up and takes over all the contracts and is not subject to the order. Lets hope not. I suspect it has to get resolved at a higher level with a change in government policy somehow. We'll see.

IPv6 will get to be even more important - good chance RIPE run out of IPv4s this year and that will be scary for a lot of people.

I hope I get over this bug of the last few weeks, Norton claims to be "anti-virus", but does nothing for a tummy bug, shame (I should sue). Except everyone seems to have had it with blocked nose, sore ears, sore throat, upset tummy, shits, headaches, and everything else a minor illness can throw at you for weeks on end. Not funny, but it better get lost as I am getting fed up with it now and so is everyone else.

Personally there is a good chance of insulin injections this year too - with a mother than has be diabetic for around 50 years that is not a huge shock, and I think it will be a relief to get off these various pills they have me on now.

The games machine (w i n d o w s?) is working, but heats up my study even more - still racing someone to get a Wogen to level 85 though I am now playing in 3D which is very cool. TBH I am really not that keen on SWTOR - it is just WoW with light sabres. Oh, and that is bloody con too - you buy the game with 30 days game time - but it will not let you use it unless you either sign up a bank card or pay more pre-paid game time, and your 30 days gets used up while you decide. Not on! A con! I will remember to cancel my card! As an aside, that policy makes it a shite gift to give someone (I gave a mate a collectors edition pack). Really not good. Sorry Mike.

I got a few pressies right - apparently "Badass Lego Guns" (the book) went down well with Ian and his kids. Certain bottles I got people were right too - phew.

Personally - I had very few pressies. I got some nice cuff-links, a WoW headset, chocolates (good for a diabetic?), dalek jellies (fun), and a few clothes. The problem is that Christmas can so easily not be about giving. I wanted some more shirts. I normally get good quality heavy 100% cotton shirts. Told not to get any as "Christmas is coming"! I now have several "value pack" poly/cotton shirts that I would never have bought. If not for Christmas I would have got what I wanted when I wanted it. Now I feel obliged to wear something I find uncomfortable. Next year, I buy WTF I like and if someone gets what they think is the same, tough.

That said, you also have the whole measuring up what someone spent. I spent thousands on a certain person (who earns 10 times what I do) - I got cuff-links and clothes. But that's life.

What is nice is that we had no fights on Christmas day, or New Year's day (or in-between). We had all the immediate family over on Christmas day even. The young one (Bobby) got loads of pressies and had great fun. We even had a nice Chinese meal last night. It was, overall, a pleasant holiday week.

So 2012 - bring it on!