2009-09-30

We're too efficient?

When we report a fault on a broadband line to our suppliers, wherever possible, we try and identify a clear quantifiable test criteria which we can test.

This means that when our favourite telco send it back "for retest" we can carry out the retest automatically, confirm if fixed or not, and reject the clear back to them if the retest fails. We make a point of clearly stating the test we carried out, when we carried it out and what the outcome was whenever clearing the reject.

Obviously we also monitor manually and escalate and so on as necessary, but our systems can handle the basics 24 hours a day.

The numpties only have the cheek to complain via the account team that we have some automation clearing rejects and they do not like it!!!

Rather than address the real issue - that they send back faults without actually fixing them, sometimes hundreds of times, they are actually complaining that we are efficient in performing the retest THEY ASKED US TO CARRY OUT...

Talk about losing the plot.

They clearly run the fault desk as a game - getting points for rejecting faults quickly, etc, rather than actually trying to resolve issues. And our efficiency upsets their high score.

Needless to say I have sent in an official complaint at the number of times they reject faults without fixing them, and have a list of example faults and number of rejects which was, err, copy/paste from their complaint to us!!!

It is a waste of time I am sure. But who knows.

Airport WiFi!

OK, so airport lounge has free wifi, as it should.

Only trick is that they seem to be lacking a DHCP server at the moment!

A quick tcpdump shows a lot of NETBIOS packets from 169.254.x.x and lots of unanswered bootp requests. Ho hum. A lot of puzzled faces curing at their laptops too.

Thankfully it was pretty obvious what network was in use (192.168.0/224) and gateway (192.168.0.1), and the hotel's DNS servers still in my config worked as well. So I have wifi!

How hard can it be to get things like this right FFS.

Card fraud protection is not protecting ME

As expected, one of my cards has got blocked this holiday by a bank's fraud protection.

OK, I can understand why they do it. But I wish they would be honest about it. I spoke to my bank manager and he said they can turn this off but then I would not have any fraud protection on my card.

What they seem to forget is that is someone pretends to be me and convinces them to part with money it is the bank that have been defrauded and not me. Any inconvenience I suffer is a side effect of that and down to the bank's gullibility. Nobody fooled me or lied to me - someone fooled the bank. I was not involved.

What winds me up even more is that when it is me that is defrauded, e.g. when some company failed to put through a refund, the bank were not interested in helping at all.

They are protecting themselves at significant inconvenience to me.

Now, if they did send transactions and authorisations as they happen then I could tell them if their was any fraudulent activity when it happens. They could even have a query mechanism by some means to double check any they thought were suspect. A little bit of sensible use of technology would avoid all of this inconvenience.

What I really want from a bank!

I heard an advert on the way to the airport coming here along the lines of Wouldn't it be nice if your bank did other stuff like roadside assistance or insurance, etc... which kind of winds me up.

What I really want from a bank is to manage money. To actually send me details of transactions and authorisations by some means (such as XML SOAP) as they happen. To send full transactions details for BACS and fast payments with the sort-code and account number and payment reference. To allow me to simply submit digitally signed payment and transfer requests (XML SOAP again would be fine).

Why can't any banks just get this right. A queued transaction posting system would be a lot easier on their computer systems and firewalls than their existing on-line banking systems.

How hard can it be?

I have tried asking banks and got nowhere. Maybe we should set up as a bank (not one that lends money).

2009-09-29

Back to work soon :-(

Well, it has been a fun holiday, but I head back tomorrow. We're off to Raffles tonight as everyone says you can't go to Singapore without seeing Raffles!

F1 was fun... Got a few pictures.

Much work to do when I am back...

I wonder how our favourite telco have been getting on. I get the impression that they are trying to sort the major issues with the new network, albeit slowly. Not quite so many major screw ups the last week.

2009-09-28

Lions and Tigers and Bears

OK, if you are ever in Singapore I do not recommend the famed Night Safari...

It is probably not too bad as something to do at the end of a day at the Singapore Zoo.

Basically it is a tram ride around the catchment area in the dark (well, duh!) and yes you do see many varieties of animals (including lions, tigers, and bears). But the overall effect is not unlike a ghost ride with dimly lit alcoves left and right with stationary animals just about visible and an inane commentary from someone that sounds like they do voice over for the discovery channel, badly.

It is not really anything like a safari and not a patch on Longleat.

2009-09-26

Moving goal posts

One of the fun things about a rate adaptive ADSL service is that, unlike fixed speed services (500K, 1M, 2M) an adaptive service can, err, adapt!

This means that when there is a fault, instead of just breaking, a line can often adapt to a lower speed. Sometimes much lower.

Of course there are also cases where speeds change anyway due to changes in line characteristics. This can be short term and long term variations. Other lines in the same bundle using ADSL, etc. So some variation is part of what is normal for an adaptive service.

So, if you are sensible, you work out what a new line can do, a maximum stable rate (MSR), and then you set a fault threshold (FTR) below which a sync rate is considered to be a fault based on a straight margin below the MSR.

This gives a nice clear defined rule on the allowable variation of a line sync rate and divides normal from faulty nicely with no argument. A line has been drawn and everyone knows where it is.

Of course, if you are a major telco, you may find this rather inconvenient, and think that maybe it would be a lot easier to just move the goal posts rather than fix the fault. Doing that manually was bad enough, but wouldn't it be even easier to move it automatically when a fault is reported.

Then, of course, you make sure that the agents on the fault desk think that is normal and have never seen the contract or handbook which clearly explains the FTR is set for the life of the service and that sync below the FTR shall be investigated. No, far better to have long arguments with ISPs until the give up.

Of course, for added fun, make it so that if you report a below threshold fault you send engineers that do not understand and just check there is sync (at any rate) and report as right when tested so you can charge £160 for a special engineer (even if a special engineer was not ordered). Just to rub salt in the wound you can have the engineer report what sync speed he got, being below the threshold, and still say right when tested.

Arrrrrrrrrrg!

2009-09-25

Hotels

Is there ever a hotel that actually just works. We've had the internet in the room broken several times, no milk for coffee, flat batteries in TV remote, etc, etc...

The one thing that does work is we get free cocktails 6pm to 8pm in the club. Of course you get free cocktails but they will charge us extra if we drink the bottle of Evian in the room! And then, most of the trip, we'll be at the F1 at that time of night. Oh well.

Had a fun evening last night - one of the locals that a friend of ours knows took us to little India, saw a temple, and all sorts and had a really nice meal...

Should be an interesting few days...

2009-09-22

Pissed off

Just as I am about to fly out, and I see a certain major telco has just changed their billing without any warning at all. They used to bill the previous month and suddenly they catch up to billing current month so on a bill costing many tens of thousand a month I get a bill a few minutes ago that is two months in one. Bastards is all I can think. They could have at least warned me. It is not funny. Good way to start a holiday... Humph.

2009-09-21

Marketing!

One of these days I'll have to do a proper course on marketing. It always confuses me.

Lately we have been very publicly berating one of our main suppliers. It is largely in an effort to get them to improve services as normal channels are not getting things fixed (and there is a lot to fix). We won't put up with a crappy service from anyone, even if that means changing suppliers after many years. But we do try and get things fixed...

What gets me, is all this doom and gloom explaining how this BRAS is broken today or that exchange is still congested, etc, seems to be getting us way more customers.

I'm not complaining. I always thought being frank and honest was good for business, but it still surprises me sometimes.

One of the best bit of marketing we came up with is offering to fix faults people have been stuck with on other ISPs. I don't think we have had to pay out on that once and the comments on thinkbroadband are great. I think Shaun is becoming a legend...

I'll get the hang of marketing one day.

2009-09-20

72

Yah, level 72.

I used to get the top-up cards for WoW as I do not play it very often. That way I would only pay when I play, well, for 60 days anyway. When I finally get it on a debit card so they charge automatically I just have no time to play at all - maybe a few minutes at a weekend!

Anyway, I am officially on holiday and actually managed to play for maybe an hour and finally leveled. It will take me years to get a character to level 80!

Thunderhorn, alliance, gnome mage, AAISP guild, RevK.

2009-09-19

Thermostats

I give up, is it just me or what.

I know how a thermostat works, it is simple. You don't have to know the inner workings. The logic is simple. Set the temperature you want. For heating systems it turns the heating on when below and off when above. The heating is only ever on or off. There is no extra on mode that heats up quicker.

So why is it that certain people (who shall rename nameless) cannot get this concept in to their head. Quote from one daughter "It was hot so I turned the thermostat down"...

But but but, it was like 29°C and the thermostat was set for 20°C so heating was off. Not surprising as it is the middle of summer and so had been off for months, and she turns it to 5°C... Like that is a sensible temperature. She expected to set it to something higher, like 30°C when it gets cold, and so on?!?

It seems that many people use a thermostat like an on/off switch. Set low to turn on, and set high to turn off (for heating systems). They just do not understand.

Its like a certain other person that insists on pre-heating a gas oven by setting to gas mark 9, and then later setting the right temperature. I think she thinks it gets hotter quicker if you set to a higher level.

And what drives me even madder is that you cannot explain it. Well, you can, but it does not actually help. They still do it...

Am I alone in finding this very annoying?

2009-09-18

Police with no clue about email?

I see it on vans on the motorway, a van with big sign writing, lets say Company Name and a web site such as www.companyname.co.uk and then you get the email address companyname@aol.com or companyname@btinternet.com or cn1024@hotmail.co.uk.

WHY?!? They have the domain after all, why not info@companyname.co.uk ?

As if that is not bad enough, I was just handed the business card for a police officer. Lets call him PC Fred Bloggs to save him some embarrasment.

PC Fred Bloggs
Police Constable

Bracknell Police Station
The Broadway
Bracknell
Berkshire RG12 1AD


Tel: 08458 505 505
Fax: 01344 823 419
E: Fred.Bloggs@thamesvalley.pnn.co.uk
Web: www.thamesvalley.police.uk


Talk about unprofessional!
They have a nice domain thamesvalley.police.uk. Why on earth is it not Fred.Bloggs@thamesvalley.police.uk

And worse, pnn.co.uk is actually owned by the ISP (Commerce Internet Ltd) not even the police. So not registered and managed by an ISP whilst owned by police, no, actually owned by the ISP as the registrant.

And worse still, the ISP uses uk.net subdomains themself (major fail for an ISP) and their name servers, but the name servers are AWOL. So the email won't work.

DNS check results for thamesvalley.pnn.co.uk:

16.7% of queries will end in failure at 193.111.200.7 (ns0.cerberusnetworks.co.uk) - failed to resolve ns1.commerce.uk.net due to 193.111.200.7 - no such domain

16.7% of queries will end in failure at 193.84.87.10 (ns1.cerberusnetworks.co.uk) - failed to resolve ns1.commerce.uk.net due to 193.84.87.10 - no such domain

16.7% of queries will end in failure at 216.218.244.58 (ns2.cerberusnetworks.co.uk) - failed to resolve ns1.commerce.uk.net due to 216.218.244.58 - no such domain

50.0% of queries will end in failure at 213.219.35.202 (ns2.commerce.uk.net) - query timed out

Why can't people use their domains

You see it on the motorway - a van with huge writing on it, and company name on the van. They have a web site www.companyname.co.uk and then you see it... companyname@aol.com or companyname@btinternet.com as an email address.

Why not info@companyname.co.uk. They have the domain after all... Why stick your self and some big expensive sign writing to just one ISP?

Well, it gets worse. I have just been handed a business card from a local police officer. Call him PC Fred Bloggs for now...

So phone number formatted sensibly, as is fax number. Good start.
Website (Web: on the card) www.thamesvalley.police.uk
Email (sorry, just E: on the card) Fred.bloggs@thamesvalley.pnn.co.uk

Why?!?!?!?
Why not Fred.Blogs@thamesvalley.police.uk

pnn.co.uk is an ISP (Commerce Internet). It is not even registered to the police!!

To me, that is so unprofessional. You expect the police to have some sort of control over their own email.

It is not +44 (0)207 123 4567, it is +44 20 7123 4567

Just a pet hate but I have to say it some time, so why not now...

I see people with +44 (0)... on business cards, even for telcos that should know better.

The + you see on international phone number is specified by a standard. An international standard called E.123 and it defines the formatting for national and international phone numbers and email addresses and web sites. Such as you would use on a business card. It is not a big standard, so why use one bit and ignore another?

If you think +44(0) makes sense or looks cool READ THE STANDARD!

7.1 International prefix symbol

The international prefix symbol should be + (plus) and should precede the country code in the international number. It serves to remind the subscriber to dial the international prefix which differs from country to country and also serves to identify the number following as the international telephone number.

Simple. So people follow standards and show numbers starting +. So far so good. But...

7.2 Use of parentheses
The symbol ( ) (parentheses) should be used to indicate that the digits within the ( ) are not always dialled. The ( ) should enclose the trunk prefix and trunk code in a national number, and the trunk code when the trunk prefix is not in universal use within a country.
This is done to remind the user not to dial the enclosed digits for calls within the same numbering area.
The ( ) should not be used in an international number.

So, there you have it. You do not put (0) in an international number or any parenthesis.
Indeed, that definition is pretty clear that (0) means that you do not dial 0 when in the same numbering area (UK) and do dial it when not (outside UK) which is exactly the opposite of what you need to do!

Why the hell do people do this. Its a meme. One business card had this and instead of doing it right people just copied that and assumed it was the right thing to do. It is not.

Consider the options:-
  • UK caller that does not understand + notation: will find +44(0)... even more confusing. However, can look in phone book, check with operator, etc if it is properly formatted without (0) and find out.
  • UK caller that does understand + notation: (0) does not help them and may annoy or confused them even as non standard.
  • Non UK caller that does not understand + notation: Not helped by (0), and without it they can ask operator, etc, and find out as it is a standard. No idea if to dial 0 or not.
  • Non UK caller that does understand + notation: Does not understand (0) as it is non standard and so does not know if they should dial the 0 or not. If they understand UK numbering or have something similar they may guess but (0) does not help that.
  • GSM user: can dial + notation as written if standard, and (0) does not help
In no circumstances is anyone helped by (0) in an international number and in many cases it causes annoyance or confusion. It makes you look stupid!!!

Oh, and of course, the area code for London is 020, written +44 20 .... and not 0207. 0207 IS NOT AN AREA CODE! You can tell this as you can call from one 020 number to another 020 number by dialling the 8 digit number after the 020. You cannot call from one 0207 number to another 0207 number by dialling the 7 digits after 0207!

Similary, and local to us, 01189 IS NOT AN AREA CODE

What gets me is that the non-standard use of +44(0) is spreading fast. It is like a rumour. You see it everywhere. Why on earth is the correct standard not so good at spreading?

Anyway, I hope you will all use this as a chance to be technically correct (the best kind of correct) and one-up anyone giving you such a business card... Have fun.

2009-09-17

Watchdog?

It must be a bad state of affairs when you watch Watchdog and find yourself agreeing with the evil corporation and not the consumers. Maybe I have been running a company too long :-)

They were basically saying PS3s were dying, after warranty, and moaning that Sony wanted a fee to repair them.

Even if you go for the view that something like a PS3 should last more than a year the argument (any contract) is with the reseller. The reseller may well have a contract with Sony that says after a year they have to pay for repairs. The contract with the reseller may arguably not restrict working PS3s to only one year but that would be a separate argument. The end users have no contract with Sony and can, contractually, not expect anything at all from Sony, I am sure. If anyone should moan at Sony it would be the poor reseller stuck in the middle.

You would think Watchdog would know that - but instead they go around with PRAT (Playstation Repair Action Team, IIRC) and are repairing them. Agreed, some indication of a reflow issue and re-ovening the boards helped. Even so they admitted many that they apparently repaired then broke again. So great, if watchdog want to repair everyone's Sony PS3 for free, great. Oh, no? They have to pay the people that do the repair and maybe they don't want to fix thousands of out of warranty PS3s for free? (a tiny fraction of the number that have been sold)... Oh!

Maybe I am getting old, but they really missed the plot IMHO. And I am a big one on consumer rights. It is really strange disagreeing with Watchdog.

Rant...

Holiday mood

OK, I am getting in to a holiday mood. It may not been until Tuesday that I fly, but kind of winding down. Probably means I'll spend even more time nattering in irc over the next few days!

Makes a change not taking the kids...

Is 600Kb/s good enough for broadband?

It would appear that, according to our favourite telco, it is quite acceptable for a line to only get 600Kb/s over their network even when the line syncs at 8,128K and even when their ISP is quite happy to provide more bandwidth and even though they charge the ISP around TWENTY times the going rate for transit to get the bandwidth to the exchange.

We had always assumed that the stupidly high cost of bandwidth over their network would mean they could afford to upgrade links as necessary. After all, upgrading a link means more traffic so more money for them. Apparently not! That would be far too simple a business model.

So clearly no chance of getting everyone in the country up to 2Mb/s minimum if a major telco can't be bothered to put in more capacity where needed even when the lines can take it.

It will be interesting if ever the internet gets proper standards for the quality of what is delivered to people's homes and offices, like gas, water and electricity. Maybe then we will have standards for loss and latency and throughput.

New URL for my blog www.me.uk

OK, I think google blogspot have a bug, so I have had to do a work around.

It seems you can have a blog on your own domain, e.g. www.example.com, and by setting a CNAME on that and A records on example.com it should work as either http://www.example.com/ or http://example.com/

Problem is the domain for my blog is www.me.uk

This is great, except they do not allow you to use a naked domain like that. So I make revk.www.me.uk and set up the CNAME for that and A records on www.me.uk as per their example.

Sadly that just left a page not found on www.me.uk which is probably google blogspot removing the www part and then looking for a matching blog, duh. I cannot find any sane way to get a fault report in to them.

So, now, www.me.uk is a simple redirect on a web server.

Of course, for fun, try just me.uk as my blog and most browsers add the www on the front :-)

The new URL for my blog is www.me.uk

Money making scam :-)

OK, government trying to help the music industry by making it possible to send warnings to ISPs customers... Annoying at best. So many things wrong with their proposals I do not know where to start.

But it leads to interesting things. They seem to propose sharing cost of sending notices, i.e. ISP charges half their cost (I assume), but LINX's response (sensible, I might add) suggest the complainant just pays a fee. They go on to say the fee should also discourage frivolous complaints, i.e. be some reasonable charge. Sounds good to me.

But then you have interesting posibilities. People already poisen trackers and do things like making an hp printer accused of file sharing... How hard to set up trackers that deliberately make whole swaths of IP space apparently guilty of file sharing.... Then automated complaints go out, and the ISP charges.

So A&A customers could do this knowing we are raking it on the warning letter fees. Cool.

Maybe that would encourage them to actually investigate their allegations more carefully and actually follow the normal legal processes... I am sure we can set up some IPs for "Mr Coffee Machine" in unit 4 so that we can legitimately deliver the letters and let people know what IPs they are...

After all, it is not as if copyright is not already getting some serious special treatment. Many rights that people have are enforced by civil action only (like patent). Copyright already has some activity criminalised, something patent holders would love.

After all, if someone wasn't going to pay for a copy of something, but makes a copy anyway, then the copyright holder has not actually lost anything financially at all by that, and full recompense for the action could be done by simply deleting the copy.

In my view they need to come up with commercial models that mean the easiest way for people to get music and videos is the legitimate way and not a rip off. Then everyone is happy. Filer sharing would not happen (or if it did, it would in effect be free advertising leading to sales later).

We don't need any more special legislation for copyright holders.

Note: this article is copyright Adrian Kennard. I grant a free licence for anyone to copy it as long as you copy it whole with this note, and as long as you are not in any way connected with the music or video industry. Transferring it from this web server to your computer for viewing counts as copying. Breach of this will result in a county court claim to recover the fee I charge such people for a copy which is £1,000.

2009-09-16

Arseholes

OK, some bunch calling themselves Call Value are calling numbers that are in the TPS selling cheap calls. They are calling us a *lot*! They call withheld.

So far I have established that they claim to be Call Value and when pushed will say they are Call Value Ltd, which does not exist. At one point they gave a different name which was in adminstration but I forget.

So, playing along I have established they need you to have a residential line with a reasonable level of calls. Then they can sell a service to save money.

So next time they call we pick someones residential lines and get it signed up. Then we can get bills, but we can cancel under distance selling anyway and many other reasons.

Then finally I can report to the Police and ICO. Criminal offences of calling numbers listed in TPS and Fraud (saying they were Call Value Ltd when they were not for financial gain). And I can send them a bill for my time.

In the mean time, I'll see how good our carriers are at withheld number traces!

Grrr.

2009-09-15

The basics of broadband fault diagnosis

How fucking hard can it be?

There are two main types of fault on a broadband line. Probably as common as each other but very different solutions.

1. Copper...
The line itself can be an issue.. Sync low or wrong. Packet loss. Dropping. Etc. Even end user router screwing up.

2. Back-haul
BRAS wrong. PPP not working. Latency (cannot happen on line itself). Exchange wide loss or latency. etc.

It surely is a basic part of training for anyone on fault repair to work out which applies. One needs engineers to end user site or changing copper pairs or whatnot. One needs people on consoles checking configs, and people fixing fibres, and sorting bugs in the system.

They have no bloody clue - they will harp on about special engineers until they are blue in the face when the issue is blatantly not a sync/line issue.

Arrrrrrrrrrrg!

My latest comments on a fault:-

"We do not need an SFI as this is not a line sync issue - its an exchange/back-haul issue. Why even suggest an SFI. That shows a fundamental failure in you understanding. Please pass to someone who has a clue how thinhw work."

"Why would an appointment be needed - thing about it. If an SFI is not appropriate then why do we have to make an appointment. There is no line/sync issue. Its back-haul. Pass to someone with a brain please."

[sorry about typos]

Arrrg - google is broken

OK, this is getting annoying, and anyone that knows how to contact google let me know - the help pages are a maze of twisty turny passages all alike.

The blog is now on http://revk.www.me.uk/

I wanted it to be on http://www.me.uk/ but the do not allow on "naked domains"

However they do allow the naked domain to work with A records, so example.com would work as well as www.example.com. Great.

So www.me.uk has the A records, but does not work. My guess is a bug in google seeing www and stripping it to find the blog. But as I cannot say me.uk is my domain (it is not) I cannot make that work.


What is going on

OK, I started this blog to rant, and what happens, another day with no major issues. How can I have a good rant? This is not good enough! grrr... No, wait, err that was something to rant about. :-)

And the dynamic duo are getting to the bottom of RADIUS issues causing delayed connect and stuck sessions. That may be sorted soon. They are testing a work-around on one realm tonight. It is looking promising.

But more fun - this blogging lark is spreading. Shaun - escalations manager - is starting a personal blog. He probably has a lot more rants than I do.

Profitting from faults

Well, this is an issue that we have raised over and over again and winds me up a lot. A major telco trying to actually make money from faults, rather than actually trying to make a service that does not have as many faults.

However, we are having some fun with this (for some definitions of fun)...

Background: They used to fix faults when things break, including, if necessary, sending an engineer to fix them. No option to charge us for that even if fault was customer caused (apart from missed appointment). We try and be sure not send them on wild goose chases, but apparently other ISPs were not so careful. So they created a special type of engineer as a new optional service to check out the line and the end users kit, charging for the latter. Unknown to us they planned to replace normal engineers with these special ones. The special ones cost around £160 for checking the end users kit.

So, when we don't need an engineer, they will randomly send one without asking or arranging a time, and then try and charge us for the engineer or a missed appointment. That is annoying.

But when we do want an engineer (or rather, when sending an engineer is the right thing to fix a fault, as all we really want is the fault fixed) they refuse to send one saying we have to buy their special engineers. They also make not finding a fault their side count as must be customer kit and charge us even though they did not actually do any work. We have to spend hours each day arguing to get them to fix what we have already bought and paid for (even if that means them sending an engineer) and then hours arguing the bill later.

Mind you, the arguing the bill bit is a lot easier now - we just say "We did not order that service - show us the order from us or take it off the bill" and do not get involved in disputing clear codes or other nonsense. Nice and simple and we get to just take the amount off what we pay. They try insisting we have to do things and contact other departments to resolve it, and we just say "show us the order". They insist it will not get resolved otherwise, and we say "it will, leave it 6 years and it vanishes under the Limitations Act - we can wait"... They don't like that.

However, on to the latest fun...

What we are finding is they try and push their special engineers on us, and we decline. So they then, eventually, stop trying to do that and change their tack to "send an engineer to exchange", or "pass to another team for further diagnostics" or one of a number of other fault remedy actions that are not sending an engineer to site.

Now, in our view, that is clear proof that the suggestion of an engineer to site is not as a result of it being the correct action to rectify the fault, but as a sales pitch to sell the special engineering service. If it was the real solution they would not change their tune when we decline that service but would send an engineer to site anyway.

So, we are raising complaints about hard sell sales pitches taking up our time. We plan to start calls with "you're not going to try and sell me anything are you?" and "good, because we charge for sales calls"... Then when they do the pitch for special engineers we'll explain that it is a sales pitch now and where do we send the bill.

I may go as far as sending a bill soon. We have told them we charge for sales pitches on fault calls now. That should be, err, interesting.

7 days to go

Well, holiday in 7 days. And I should not worry - we have a good team to cope while I am away, and it is not like I'll be totally out of touch :-)

2009-09-14

Phew

OK, 6pm and nothing broke. It seems wrong somehow. You expect something to break.

OK, OK, one exchange is showing congestion and I have had to report that to our favourite telco, but that really is getting routine now.

What cheers me up is *my* code did not break - not that it is all my code in the LNS. Its rare for things to break but when it goes two days in a row (Thu/Fri) you kind of get paranoid. Running 3 more days after that with no issues kind of makes me happier that it was a fluke. Thanks to sterling work by Cliff over the weekend we have new Ethernet code for loading tomorrow that should detect and react if ever it happens again.

Still, if gets me how chips can have so many bugs these days - and I bet this is one. We'll find out, but who knows it may be another 6 months before it happens and this time just recovers pretty seamlessly...

Seems like the dynamic duo are getting to the bottom of some of the RADIUS issues too, so that will be fun. Its all happening.

And my sister-in-law is still counting down to when we all go on holiday next week... She is proper so excited like the Disney advert!

The good guys

I know I called this "rants", but there is always room for surpises and praise.

It is worth mentioning that my favourite telco does have some people with serious clue. The dynamic duo are right at the top of the technical tree and get things done.

They have been in touch this morning to say that they can see the issue on one of exchanges where we reported problems. Even though the fault desks bounce back saying congestion in the internet, blah, blah, these guys have found there is a problem. It is causing around 1% packet loss on all lines on the exchange which so many people there would say is not a problem, but it effects performance badly even at that level.

I do feel sorry for them though. They work hard and silly hours and they are in a coporate machine. I used to have a real job once and it can be a real battle. They spend their time fire fighting issues.

So, well done guys.

Arrrrg




OK, I should be calm, but this is so taking the piss it is not believable.

I have lines on an exchange that have HUGE packet loss and latency.

The stats are:- [sorry, google does not like spaces for some reason]

+---------+----------+
| rejects | count(*) |
+---------+----------+
| 3 | 1 |
| 43 | 1 |
| 99 | 1 |
| 102 | 1 |
| 104 | 1 |
| 111 | 1 |
| 113 | 1 |
| 115 | 1 |
+---------+----------+

Every time we have confirmed that the problem still exists and clearly tested the line, e.g.

"Peak latency BETWEEN LNS AND END USER (so not wider internet) is currently 235ms, which is not acceptable. Re-test criteria not met. Please investigate further. No, we are not requesting your SFI engineer service. If you need further clarification call the ISP contact listed. Thank you."

We have had some that say (hundreds of times) "Slow speed is probably due to internet congestion" which clearly indicated that they do not read what we see as the test is between LNS and EU and so not involving the wider internet.

We have had some that say "EU not Logged in". They have to be for us to measure latency. That is silly!

And, of course, I have added notes manually several times to explain the matter.

This is now CORPORATE STONEWALLING. They are just refusing to fix the damn faults.

They would rather make a process to systematically ignore their customers than actually address the issues. Well, I am not standing for it.

2009-09-13

Another long day

Well, we have a couple of exchanges with issues, and of course the faults are just being bounced back saying that there is no problem and the low speed is probaby congestion in the internet. No actual work done to fix the fault. Ho hum...

The fault that sparked all this has the BRAS corrected, probably by fluke. But the engineer being sent to the exchange turned up at the customers house un-announced and tested the line - so that will be us arguing a £160 bill or an engineer! The poor customer still has low throughput even with the right BRAS so we are now having to try and confirm a back-haul congestion issues (of which we have seen many)...

I have been pondering issues. When dealing with such a big company via the normal channels such as the account team and CRM then we are small. Don't get me wrong, we have a good team now. But they work in a part of the business dealing with small ISPs and we don't count when it comes to fixing big things.

However, when normal channels fail us and we have to resort to public embarrassment then we are as big as we are loud... And we are loud. We get noticed. So then things get to director level and someone notices the problems exist.

We had this with IPv6 issues. We raised them for months and finally published the statement they gave us saying they do not support IPv6! That got to director level and they have changed their tune.

So, either they find a way to give us a better voice within their organisation or anything big will always get to the public humiliation stage, which is not good for any of us.

As it is, my understanding is the major issues we have seen may get to director level some time tomorrow due to some convoluted connections, help from a customer, and (of course) my frank blog posts.

We'll see what happens.

Maybe I need to do some consultancy for them...

Never thought it would happen

I have never written any sort of diary, so WTF am I doing here? And what kind of word is blog anyway...

Basically, I had quite a bit to drink last night. Not unheard of, sadly, and I know I shouldn't... But that did not stop me trying to help out customers. We only do 9-5 Mon-Fri support, but we all chip in on evenings and weekends and if there is anything major and I am no exception.

Its a good idea to work at the coal face a bit when running a company. It lets me see when things are not right, not only with our stuff (most of which I wrote) but with our favourite telco.

I just got so pissed off with them when I reported something so simple like a BRAS on the wrong rate and they decide to send a damn engineer I posted that they are fucking idiots on the company status pages...

I kind of realised that was probably not such a good idea, and the post was updated pretty sharpish and it has now been toned down a lot. I suspect we'll get some flak from them this week. I know some senior people there have seen it and are very concerned that they have managed to drive an ISP to such comments. Hopefully they will realise this is the end of a long line of frustration and annoynace.

But, what can I say. This is not out of the blue, and it really winds me up. I have spent a lot of time writing systems to fight their fault systems and policies and idiocy to get faults actually fixed. It just should not be necessary. I really wish we could work together and get things done properly. Drives me mad.

Anyway, that's why the blog has happened. Somewhere for me to post and have my say.

I hope it is not boring...

My own opinion...

There are times when stuff winds me up. Some people know that I am "passionate" about my work and there are times when I express that on the company status pages.

Sadly this is not really politically correct. The company should be professional...

So this blog is for my personal opinions on stuff. It will, of course, include shit that happens at work I am sure.

Obviously, if I offend someone (even if they are a major telco) I'll be happy to post their comments as a follow-up. Obviously if it turns out I am wrong on something I'll publish an apology and even delete the original post. However, if stating my opinions it is hard to be wrong. If my opinion happens to be "[major-telco] have lost the plot" then that would be a true statement of my opinion... An apology would where I have some facts wrong somehow... I believe people are entitled to their opinion.

So, views expressed here are not those of the company...

Fencing

Bit of fun... We usually put up some Christmas lights on the house - some fairy lights on the metal fencing at the front, but a pain as mean...