Bad call

When we are setting up new servers and services we often set them up in the rack in the office first. It has a fibre to the data centre so is well connected (in fact we can test on the data centre LAN so no config change when we move). We then move to the data centre later.

The new call server is in that stage, and we were going to move it. We did not move it last week. We figured "The data centre has had as many power outages as the office so no made rush just yet". Bad call! Sorry...

Well, a certain son of mine, who shall remain nameless, felt it was a good idea to plug in half an air-con unit that had "THIS IS DEAD" in huge red letters on it, and then wondered why it all went dark suddenly. It was not even a whole air-con unit, and yes it should have been disposed of already.

So, I suspect we'll be moving the call server to the data centre soon. Ideally a Sunday morning. We'll also be disposing (properly) with the dead air-con unit and looking at "procedures" for removing the plugs of anything like that in future!

Obviously, the call server is being deployed as a pair of servers. We have to try and address how we handle seamless failover on this as some phones get upset with calls from an unexpected IP address. This is all part of what we are trying to make work now.

The good news is calls in progress should have been fine, and also not charged for.

What with our favorite telco being a pain, this is not a good day.

Refusing to fix faults

The latest from our favourite telco...

They are now sending fault reports back for "retest", and then refusing to accept our reply that the retest has failed because we have not booked their special investigation service.

So, what do do now.

• One option is book an engineer for a week's time, reject the clear and then cancel the booking. Automatically. Every time. That could be interesting. I wonder how many engineers will turn up after being cancelled. I wonder how many we will be charged for.
  
• Another is that we communicate to them that they have a fault and need to fix it by some other means - perhaps by fax to their head office - for every clear reject we do.

Of course, we also have several thousand lines that they are basically refusing to fix as they have high latency and packet loss. I think I need to report those as faults now. It will be interesting as they want us to book SFIs for those as well even though the issue is in the back-haul which is outside the remit of their special engineers.

At the end of the day we have paid them for a working broadband service. If it not working they have to fix it. If they refuse to fix faults they are quite simply in breach of contract. It is just not good enough.

I am not happy.

OK iPads are cool

That is all

Chip & signature cards - Currys are plonkers

OK, I have a chip'n'signature card. i.e. has a chip in it so that it can be properly verified as present and not a clone, but takes a signature not a PIN when you buy stuff.

I like it because I am happier saying to the bank "show me the signature" if ever there is a dispute. It is harder to copy my signature than see and remember my PIN. And it is not what any fraudster would expect so less likely to be targeted. All in all better security for me.

Most places are slightly puzzled but cope. Occasionally they have to figure out where the printer is to get it to print the bit I sign. But generally fine. Rail ticket machines just issue tickets - no questions asked :-)

Currys/PC-World were just dicks - they said they have to have a PIN. It was not until I expressed concern that surely this would give them problems with Disability Discrimination Act (chip & signature cards being usually issued for disabled people) that they made some effort.

Bear in mind that their till said "get customer to sign, check signature". They just had to so what it said.

So their first idea is phone for authorisation. But that is only for paper vouchers and their bank would not do it. They refused to authorise saying they had to use the chip.

So then they try the transax or whatever system they use to check cheques and the like and got no joy.

So eventually they decided that taking copies of ID would be OK.

Then they put it through and I sign. They don't bother to actually check the signature, or take copies of ID on the signature side.

All this to buy half a dozen iPads. What hassle.

Arrrg! Some times phone calls don't work

It is fact of life, even more so now we have mobiles, that some times phone calls do not work.

• People mis-dial, a lot
  
• Equipment mis-dials
• Connections go wrong some times
• Links in the phone network can be congested
• Sometimes you get one-way audio (rare)
• Sometimes you get audio breakup (mobiles usually)
• Sometimes calls drop for no apparent reason
• Sometimes people answer the phone and drop it and so cut you off instantly
• Sometimes phone handsets are faulty, especially if ever you spilt coffee on them
• Headsets can have the mic pointing the wrong way so you can't hear people

The list goes on.

Now, normally, when shit happens, people happly ignore it, try again, maybe puzzled by it if it is unusual, but that is it.

But if ever you "update the phone system" in any way shape or form, then every minor quirk is reported as a problem. We used to install phone systems and this was a classic feature of the initial support - explaining that "Yes, they called from a mobile and went through a tunnel so the call dropping is not a surprise. No, it is not a fault in the new phone system"...

Thankfully our customers are being relatively sane on this, phew. But it really is amazing what you can find if you actually go looking for problems!

It is also quite hard when there really is an intermittent problem to tell if it is real or not.

All good fun.

P.S. The classic one is "If I call a land line, and they hang up, the call stays connected". People are so used to mobile and SIP now they expect that when one end hangs up the call ends. People forget that land-lines don't actually do that! You can hang up, walk to another extension in the house, and pick up the call. It can take minutes to actually clear. Land-lines have been like this pretty much forever and it is not a fault :-)

*--

OK, one step closer to ditching asterisk altogether.

I now have voicemail and simple prompt and key-press on the new call server. Still have to sort fax2email, but the big thing it seems I have to do now is work around NAT installations.

NAT is evil

And this all proves how evil. To get VoIP to work with NAT you end up with the handset doing some clever stuff, the router doing some clever stuff, possibly an external STUN server to help the handset fudge the RTP, and maybe some fudging on the SIP proxy as well. Ecen then you probably don't have something that does SIP properly, but just about works in some very narrow and ill defined subset of full SIP protocol, sometimes, maybe...

NAT is evil

It is a really big bodge of work-arounds, much more so that most other things.

NAT is evil

For some inexplicable reason we have customers using NAT, so we need to reproduce each scenario. Work if how it worked with * (assuming it did in fact work), and work out what we, or the customers, have to do to make it work with the new call server. Arrrrg!

NAT is evil

Ooops

I'd just like to apologise to those that I accidentally DoSed today. A small bug in the music on hold process on my new call server started sending packets as if there were 10,000 calls on hold and caused some annoyance to our phone carriers. It was only for a few minutes. I believe we have solved this now, and we have added automated checks for any of the processes running away with themselves. Sorry.

Mad lady's house

Something of a neighbourhood intervention seems to be happening

Before

After


Through the looking glass...

Hold on

OK, now things have settled down a lot, I had the chance to do hold music. Took me around 2 hours coding this morning, but by 9am we had hold streamed from a file to multiple endpoints... Woohoo...

Took a further hour to find suitable free hold music, convert the raw alaw file to look like a WAV file (simple header), test it all, and pick which music... :-)

Now the core is stable, some ancillary projects for me to consider...

• Voicemail application
  
• General IVR (interactive voice response) application
• Land-line text application (acting as land-line text server)
  
• Fax2email application

However, I have lots of other important work, and asterisk can do most of these fine for now.

Learning the hard way...

The SIP call server is now at the stage of some fine tuning. This is the stage where we find that either the spec is not clear or things just don't follow the spec. Always happens :-)

So, things like when you get 180 Ringing, and when you get 183 Session progress, and which have an sdp payload or not, and when there is audio or not...

Turns out some carriers can send 183 then 180 then 183 and all have sdp. Some don't have sdp on 180. Some phones don't understand 180 with sdp. And so on. Now we have code that always sends 180 with no sdp and that seems to keep everything happy.

We have carriers that support Remote-Party-Id perfectly, even passing back connected line ID from the phone network !!!

We have carriers that claim not to support it, but if you use it and set privacy=full they mark the call "unavailable".

The SIP/GSM gateway they are using is damn good, and supports Remote-Party-ID, and putting calls on hold, and even tells the phone if we send a 3xx to divert the call (pops up saying "Diverting"). All damn impressive.

We are just sorting a few fine details sending calls to peopls * boxes as well now.

Oh, and we found a 4th way to transfer a call!
1. Transfer before answering
2. Answer, and blind transfer
3. Answer, hold, call someone, they answer and then transfer
4. Answer, hold, call someone, and transfer before they answer

All are handled totally differently!

Sky box is happy

Well, having fixed a few bugs, and then spent ages arguing with linux library calls this evening, we have all working well with VoIP.

The sky box was fun - or rather the supura not the sky box was the issue. Simple missing check in my code.

What is interesting is that modems and faxes over SIP are apparently bad. Well, so they say.

But SIP is providing full a-law 8,000 samples per second the same as ISDN. So, well, it should just work. But the world of SIP users and carriers have it drummed in to their head that fax and modems do not work.

Well, now we have our own call sever and no asterisk, we have SIP working better.. My Sky box will go on line and access my sky account... over SIP... woohoo.

All good fun.

OK, sorry

Yes, I moved several phones over without a proper planned works notice. I know we had lots of notices about new call server, but nothing saying explicitly when we would be trying to move lines over. It should have "just worked". For many it did. Sorry. I have been told off by my staff :-(

Sadly several people had issues and I made changes to work around some of them during the day.

What has been bugging me is the call server has crashed half a dozen times. Thankfully, unlike asterisk, calls in progress are not affected by it crashing, and it restarts in a few milliseconds - we just lose billing data. So not a major headache.

So I have been adding debug messages each time, and working through tcpdumps, and finally found the correlation. Every time one of my sky boxes at home tries to call home (via a sipura), bang! it crashes. So I have the offending message and should have it sorted in a few minutes.

Fun day

OK, music on hold was not the issue - save that for another day.

We had what looks like a bug in SIP for one of the carriers. Caused a few issues with transferred calls. Worked around for now. We'll wait for them to agree they have a bug!

We had the recording system lock up - bug found and fixed.

But otherwise all was fine until 6pm. No stale calls. Everything working perfectly.

Then my Dad manages to make and retain 5 phantom SIP calls. How? FFS this is not possible. And no, he has no idea. Ho hum.

Hold music

We have switched the office over to the new call server, and I have managed to set up the billing system to produce itemised bills with PDF and XML nicely. We have switched most incoming calls over to the new server this weekend too.

I think I can guess the next issue we'll have - and that is people knowing they are on hold.

Now, if the person on hold has a SNOM - they see on the screen they are on hold.
If the person on hold is one of our mobiles they also know, and have a nice voice saying so.

But normal PSTN calls just get silence, which is a bit of a shame. I can see I am going to have to stream some sort of hold music, beep, or message. Oh joy...

Should not take too long though.

Why can't people use BACS correctly

It is not like BACS is new, but people (and banks) continue to just get it plain wrong.

We see two key fields on BACS credits we received (apart from date and amount, obviously). They are the originator account name and the beneficiary reference. Both are 18 character text fields.

The account name is often truncated to fit and not that helpful to identify people, but the beneficiary reference is what is there for us to tell who the hell sent the money... Its a field we tell the payer to quote when making the payment. Its used for all bill payments of any sort to specify the reference for the person you are paying. If you pay a gas bill, its the gas bill company gas-account for you (a long number usually). We tell people to quote a reference that is the account with us, e.g. A12345A

So what am I supposed to do when some bozo sends a payment with :-

Originator name: MAIN A/C UK
Beneficiary reference: BROADBAND

I should set up a charity or something - the BACS victims support fund. And we should make it part of the terms that all stupid payments are in fact donations to that charity... Grrr...

Fettling

Software gets to a stage where it all works, but need a bit of fettling...

We have been deploying the SIP server today, and it has been a day or "tell me what is not working now?".

All minor things - no ringing tone when calling external numbers; call transfers not clearing on the transferring party phone; lost audio on external calls; crashing if you dial an invalid number; etc... All sorted quickly and not affecting calls in progress (well, mostly).

But over all massive progress and testing lots of the features - things like retrying a SNOM every second unit you take your phone off DND if you are in a hunt group with call queuing set. I probably should not have a means to do that to a BT number :-)

What can I say - working well.

We have all the CDR data we want and I plan to make the billing system this weekend. This means making pretty PDF phone bills for customers, and non-pretty XML files for customers with clue...

And for the techies - least cost routing in a single SQL query:

SELECT * FROM (SELECT * FROM `NumberCost` WHERE %#s LIKE CONCAT(`number`,'%%') ORDER BY LENGTH(`number`) DESC) AS subselect GROUP BY `carrier` ORDER BY `%s`,`ac`,`mc`,rand()

One SIP at a time

Well, after much annoyance my SIP server has understanding of the SDP it passes on. At least enough to understand "hold". This allows call transfers to work and not end up with one or both ends of the transferred call being "on hold". Not too bad when they are a SNOM with a hold button, but not good when it is a normal phone line calling in... Grrr.

And the gigaset did not like my accidentally sending Content-Type twice, but chose to totally ignore the message rather than actually complain... Fixed.

Anyway much progress and we now have all the features we had on the old system - some of which involves passing the call to the old system (simple IVR, voicemail, fax2email, IAX).

A voicemail app will be simple; an IVR not too hard; and fax2email may be challenging but I bet it is way better than * when I have finished.

The CDRs look like they are working, so ready to try making bills from them!

We now have the big step of switching most inbound calls in one go to the new server. Eeek. We'll try that soon...

All good fun.

Visit from the crazy lady

OK, house opposite is empty - has been for years - over grown garden, etc.

I got to see inside for the first time today, and now need a shower and some disinfectant.
Place looks to have been broken in to, but as the PSCO said, he has seen drug dens that are neater.

Crazy lady has a mental condition, or two, or three. She reckons stuff has been taken (how the hell can she tell?).

House is piled high in rubbish - paperwork, stuff, all sorts. Absolutely unbelievable!

Hopefully she can get help one day.

P.S. No cats involved, though why not is beyond me!

Junk calls

Oh, and during my testing, I would have phones on odd phone number set up on test systems with debugging screens and packet dumps and so on.

And what happens, not once but several times, those bloody junk calls. They either try and sell me debt handling or try to sell me cheap calls. They are so f'ing annoying anyway, but when they screw up my testing when I am concentrating it is even worse...

Something has to be done somehow.

Yes, I have taken to answering and threatening to kill their family. It is mildly satisfying.

*

We have used asterisk for years now. I even contributed to the code by writing the land-line SMS module.

For those that don't know, asterisk (AKA "*") is basically a phone system. It allows traditional phone lines, digital phone lines and internet telephone connections to all inter-work (if you have the right bits of equipment). It does voicemail and voice prompt/response systems as well.

However, after years of using * I am now using SIP directly. I wish I had done it years ago. In just under two weeks I have coded a complete SIP call router that meets our needs (centrex type operation, hunt groups, etc) including some special handling for the mobile service.

I have also coded a call recording SIP proxy that integrates in - it is a separate stateless (from SIP point of view) proxy that adds Via and Record-Route headers and changes the SDP to allow it to force A-law and capture the call to disk. It makes a native A-law stereo wav file and a script will convert to MP3 or OGG if needed and email. Extra headers in the INVITE say who to email. Nice and general purpose and easy to scale. Probably easy to integrate with other SIP systems.

I have also coded a simple SIP media endpoint that plays wav files, either as call progress (183) or as call content. Again, a separate module so easy to scale.

But this is just the start - now I am using SIP I discover we get more from some carriers. One even provides call divert info, so a call diverted to us shows as such and we are told what number did the divert even. This is useful proper telco carrier stuff. Just need to get the other carrier to do the same.

What it does mean is that I think we are the first telco to do anonymous call rejection on mobiles now. Well, maybe others do now, but when the law came in, in 2003, the mobiles were not interested in complying and the regulators decided to do nothing (good having laws like that!).

So, two weeks solid coding and something good to show for it. We won't be ditching * completely as it will still do IAX relaying and voicemail and a few things. But it was proving unstable and hard to scale. We have a few weeks of tinkering and testing I am sure, but I can start on some other work in the mean time.

New skin

OK - thought I would give it a re-vamp. Amazed that I have already had calls and texts commenting on it! Positive comments I may add.

Anyway, you can follow up here now...

Insisting on watching in low def

What is it with people?

It is bad enough watching TV on the low res version when there is an HD channel available.

But what the the is the logic in complaining when I change the channel to HD?

I almost wish they did a black & white version so they could watch in that instead.

Arrrrrg!

Drip drip dip

Well, we had fun yesterday.

Someone left a tap on upstairs and in the morning it was water, water everywhere...

Missed everything electrical, but flooded main office downstairs and upstairs, and toilets, and kitchen area.

Well done to Glenn and Andrew with the plumbing and the water sucking vacuum and the industrial de-humidifiers and the hard work... Now we just need to do the carpet cleaning to remove the water mark and find a few ceiling tiles...

And I was doing so well

So, being up for hours in the middle of the night is not going to help matters with getting on with coding today...

Our favorite telco screwed up and took half our lines out.

Arrrrrrg!

nggggg

OK, so I have been coding solidly, 7am to 10pm for nearly 5 days. New SIP call server, from first principles. All interesting stuff. Well, I would not be solidly coding if it was not a bit fun!

I was quite chuffed that in about two days I had the first version handling phone registrations and passing calls, but several bits have been re-coded to be more robust and tested with various phones and carriers. It is looking pretty good now.

Damn SNOM phones negotiating encrypted audio and then not stopping encryption when the call transferred - arrrrg! That wasted many hours. I'll have to email them and explain!

Another day or so of work to sort call recording, and we can start testing with customers.

I doubt I can ditch * completely as we'll need voicemail and IAX inter-working I expect. But we should be able to make a much more scalable and robust platform now...

Sadly, when all that is done I have the next big project to started (OSPF)...

Work work work...