Playing with microphones

The latest LED board designs have included a TDK PDM I2S microphone - the idea was to make sound reactive LED strips.

It is tiny (3.5mm x 2.65mm x 0.98mm) and cheap(ish) $0.76. But it is digital.

Now, this was a can of worms for which I was ill prepared. I2S is not like I2C, it seems, and there are a number of ways of doing it.

This particular device is PDM based, so has just a clock and data line - the clock to the microphone, and data comes back. The concept is that you have two of these, one set for right and one set for left channel, on the same clock and data. The left and right are on the rising or falling edge of the clock. It is fixed 16 bit per channel, and works at a specific (wide) range of frequencies of clock and hence sample rates.

But other I2S devices work in a different way, with extra pins.

I went through some stages for this...

• Simple analogue microphone (cheaper) - realised that was a daft idea.
• This mic, set to right channel, and it did not work.
• This mic, set to left channel, and it did not work.

What did not work was using WLED, a common LED driver package which works nicely on our LED modules. We realised it wanted I2S, and then realised it only did PDM on left channel, hence the different iterations, but no joy. Hopefully we can get some feedback in to that project so it does work as TDK is likely a good brand. It is amazing how sensitive it is.

My guess is it is not clocking at a rate the device will handle when using WLED, sadly.

So time to do it myself, as always!

I coded I2S as per ESP32 library, and, well, it worked, I got raw data. I could even tell it I want mono only and which channel. Amusingly using the wrong channel works too as the data floats for the other channel - though I picked up some high frequency noise that way. So best to set it correctly (I nearly said "right").

The next step was a simple FFT function - I have not gone for anything fancy or uber efficient, as it keeps up well enough, especially when I went for some simple oversampling on its input. I cannot clock the microphone as slow as I would like for normal working. But even at 30k samples/sec I could keep up with FFT 30 times a second, just, all done with floats. These ESP32s are impressive.

I have made the working audio range configurable, and averaged the FFT down to a small number of bins (24) which are log of frequency to fit better with musical scales (options for linear).

This has led to some simple audio responsive LED schemes - a simple brightness based audio spectograph - smoothing from 24 bins to however many LEDs you have in a chain. And also a simple overall volume based bar graph, and one that is RGB for three levels of sound.

Lessons learned.

Some automatic audio gain was pretty simple.

The next lesson was the log scale on frequency. That was not hard. But made an option.

I reduced the full FFT result to a small number of bands, 24.

Also I needed to make it do a peak level per band and damping (configurable) to give anything that looks nice.

The frequency range is configurable, but log based causes gaps if you go too low, so ended up 100Hz to 4kHz for now, log spread over the 24 bins from a sample rate at a typical 25 blocks/sec and actually sampling over 50-60ks/s down graded (average of oversample) to 12-15ks/s for the FFT giving results to 6-7.5kHz which is more than enough for the bands I am using.

I also ended up making the samples fit the LED update rate synchronously, typically 25 or 30 per second (configurable) to avoid any extra jitter/aliasing effects.

Oh, and when using RGBW, with a colour band for spectograph, before automatic gain kicks in you can have over 1 on a band, so I made that push in to turning on the W, which worked well.

The result

Update: I have now tweaked and got cleaner response across the range, putting 100Hz to 6.4kHz in to 48 log based frequency bins. See youtube for frequency tests.

Now for use with a live band at the pub!

And HIWTSI (my son's business) is doing LED system installs now.

One Touch Switching

It has been some weeks since One Touch Switching was fully live.

TOTSCO say over 100,000 switch orders now, so it is making good progress, well, in principle.

In practice a lot is working, and in terms of volume, with the key players, as well as the likes of A&A, all working reasonably well now, switches are happening, both ways. We are seeing things working both ways and correct billing as well, which is good.

But there are still some challenges.

• Whilst I cannot go in to details, even the big players are still facing some issues, mostly small issues but some bigger, and some with workarounds for now, and for which they are rolling out as updates. Daily calls continue with industry (yes, some I have taken from the pub).
• A lot of smaller players are catching up, but many face the challenges of the huge holes in the specifications. These are still frozen and so grey areas, errors, and contradictions abound. Only once they are live with other ISPs are the issues apparent. In at least one case we have to ask how a CP managed to be live as they had some fundamental errors (like SOR not being a UUID) that should not have passed even the very poor testing TOTSCO claim to do. Even larger CPs are not entirely agreed on some field specifications because of poor wording in one of the better parts of the spec, but are working on it. The daily calls help, but only happen with the early/bigger CPs.
• There is now a formal process for inter-CP communications to resolve such issues, but not everyone is on it yet. There is a fallback process as well. So things are happening, and some of these issues with smaller CPs are being fixed as well.

But even now, even this weekend, I have seen incorrect messages and errors. I have reported, of course, and it may be these end up as defects on the daily calls, we will see.

I worry what will happen when the daily calls stop - reporting an issue to a CP may mean they ignore it rather than spend resource investigating, fixing, and deploying. At present CPs have that resource assigned, but they will not forever.

What next? Well we keep at it.

The next big step I can only hope for is an unfrozen spec and a lot of clarifications and updates. It will be interesting to see how that process happens, and how we can be involved. I have a lot to say on clarifying the specifications and I hope I can be involved in making it happen. But every change will need a lot of agreement, and even some changes by all CPs in some cases. For now, there are some silly compromises like all strings max 256 characters (which resulted from a global update to a Swagger definition system, rather than any informed debate or formal specification change, and is annoying as tinytext in mariadb is 255 characters not 256). Even so, some agreement on even the vague magnitude of things like correlationID is a good start. I suspect, in practice, that one may get defined as smaller, like 64 characters. In hindsight it should have been a UUID, unique per message, but too late to do that now. The problem is the smaller/newer CPs are not in on that discussion, so don't know. Big CPs guessed at 36 characters (UUID size), 56 characters, 64 characters, and so on, as there simply was nothing in the spec, but most had to set something in their code. We changed our handling within the first few days as we understood how broken the spec was, and now handle any size (well, megabytes) but other CPs don't, and we have limits on a load of other strings anyway.

For now, I have every message we receive, and every message we send, run through my NOTSCO checker and reported to me. I feel it is only fair to test us as well. Over time it will only be problem messages that I need to monitor. It has actually highlighted some issues in what we were sending (where customers manually type an address, mainly - I have added more checks now). But monitoring real life message has also meant updates to our checking in the live system, and updates to my NOTSCO tools.

My latest changes include actually using the longer agreed size of correlationID to ensure we tag the message type as a (small) suffix on a UUID, so that we can quickly (pre-database connect/check) validate messages we get back are sensible and reject them. Why? Well one small CP is sending nonsensical replies to replies, or reusing correlationIDs from previous messages with different messages, both of which we can now pick up in milliseconds, and cleanly reject. It looks like they are working on it, but no actual communications back to us, which is a shame - we're happy to help and advise if only they would talk to us.

Overall - OTS is happening and mostly working, so do try it when you want to switch telco.

Pubbing it

I used to go to the pub once or twice a week, a few years ago, not so much since COVID.

What is weird is finding myself going to the pub almost every afternoon now.

OK, well, morning as well, even if that is installing a PIR, or Access Point, to CCTV camera, or door contact, or just some signs. But then in afternoon just go see how it is going - and people watching...

Today I jogged in to meet an unexpected BT engineer!

This is a bit new to me.

Another pub

If you have followed me on mastodon you will know what is happening.

Some of you may know I purchased a former pub for my home in Wales nearly 4 years ago. Not a pub now. Settling down in Wales, as are many of my children, and grandchildren (half of which are Welsh!).

But now, two of my friends have decided they would like to actually run a pub!

There is a saying: "How to make a small fortune running a pub? Start with a large fortune and work at it until it is a small fortune".

But that does not mean it is impossible, and it seems sometimes that the whole industry is run on the edge of solvency somehow. I may be wrong, and obviously there are pub+hotel and other combinations that work well, but the way some pubs are run is, err, interesting.

I have got myself involved in helping them in a lot of ways now. It started as just "we'll sort internet", but doing more and more. I'm spending most days there helping out right now.

It will be very interesting to see if they can make it work. They are organised, and experienced in related businesses. They are hard working. They take customer service seriously. I think this can work.

Lots of little things, like a defibrillator on the pub - I'm sponsoring that (well A&A are) - one of my staff died suddenly and that scares the shit out of me, more places having these will definitely save lives. And if it saves one life it is worth every penny.

And, of course, the WiFi has IPv6.

They are going for pub + cafe, with decent coffee, and choice of milk and so on, like many other "coffee shops", but also with early opening for coffee and breakfast. Do the coffee right and it could make as much as the alcohol, who knows?

But making sure they are starting it out well from a tech side is my project. They are sorting refurbishing and redecorating, and some serious cleaning. I am sorting decent WiFi/internet, and CCTV and a few other things. Even to the extent of properly registered with ICO and proper published privacy policy. I hope I can provide input on the way some of us would like a pub run. But it is their pub, not mine.

But for now - grand opening end of next week before Abergavenny food festival (only drinks for now), and then finish the refurbishment and start food as well.

I am, however, damn impressed with the WiFi we have put in. It does not just cover the front, but the whole of the Brewery Yard car park and even out the to A40. Some opportunities for take away coffee to the market traders maybe, with the working WiFi connected credit card machine... We'll see how it goes.

I'll post more on opening dates, and so on...

OTS correlation ID

It is complicated to report this as some details would be covered by ramp up rules, where I cannot provide details, but now we are in live on One Touch Switching I think I can report. Even so I will not name CPs for now. This is more about the process, and specification and the fiasco.

To be very fair the main author of the specifications is someone that I feel a lot of sympathy for, under pressure, and then under a change freeze he would not have agreed. He did his best and none of this is a dig at him or his employer. Much more a dig at the process.

Weirdly correlationIDs have turned out to be a big issue, and continue to be so.

What is a correlationID?

Basically one of the message fields in One Touch Switching is called a correlationID

The big issue is the vague specification. It is a field the sender of a message sets so they can correlate the response.

Hindsight

To be clear, in hindsight, and what I have said, is correlationIDs should be per message unique and a UUID. Simple. If the spec had said that a lot of pain and hassle would have been avoided.

Problems?

The problems are various...

• It is not a defined format
• It is not a defined maximum length
• It is not well defined when and how it is unique, or not

TOTSCO 66

So the issue is that some CPs assumed it would be per CP per message unique, and so used it to identify (and ignore duplicates). Indeed a notice from TOTSCO suggested it is used to de-duplicate messages.

A real issue is "why duplicates" which is another issue - they have to be failing to respond in 3 seconds for that, and maybe that is what they should have fixed.

There are also a lot of cases where a duplicate is not an issue, if done right.

But TOTSCO 66 said you can de-duplicate based on per CP correlationIDs being unique per message.

TOSTCO 67

The next notice back peddled a lot, and I was instrumental in raising this I think. All because the specification was so vague. The new recommendation was two fold (a) don't de-duplicate on correlationID, and (b) don't send duplicate correlationIDs. A pragmatic approach without direct blame either side.

Indeed one idea was, if you use correlationID as a more "overall message flow" ID, append or prepend something to each message so they ends up unique.

So CPs are, indeed, doing both, yay! We have all seen a lot of work making this happen, and well done to all the CPs doing this.

To clarify we went through something like three iterations to get this sensible on our systems.

Length

Oh, did the specification say how long a correlationID could be? No. It did not. Why would you say that?

Well, maybe it did, sort of, TOTSCO link to some schema thing (swagger?!) which was updated after the frozen spec and the latest version of that says 256 characters. That is mental long, and I have no clue if 256 characters or 256 bytes (they are different in the UTF8 world of JSON). Just to say, A&A can handle any length up to mega bytes, if needed.

Turns out TOTSCO had limits on what they would handle, as this is a message envelope thing. They were ignoring, and not apparently reject cleanly, if too long. I have not tested with 256 x big unicode characters, yet!

But we have a big CP that would not handle more than 64 characters, but sorted that before 12th, well done, I won't say who. It was a very reasonable choice for them, and I understand it. But well done moving to 256 characters, or bytes, in time.

We now have another big CP that would not handle more than 50 characters. Not yet sorted, but will be soon.

Why such long correlationIDs? Well BECAUSE of TOTSCO 67 notice, CPs using a 36 character UUID and adding a timestamp. That just pushes over 50. And to be honest 50 was also a reasonable design choice.

So 256 characters, is that OK? Guess what, the tinytext type in mariadb is 255 characters, FFS! If I had to make a silly long limit I would have said 255 not 256, really.

Ping pong

One of the mistakes we made at the very start, for a day or so, was assuming correlationIDs were ping ponged over the switch process (match, order, update, trigger). I had fields in the database to do this and code to do it (they were tinytext).

Why did I assume this? Well the specification did not say, but the test cases did, they had correlationIDs on an OrderRequest following on from the MatchRequest. They looked a lot like they should ping pong over the process.

Well I worked it out, but did every small CP that is live today?

The answer is no, they have not, and at least one small CP (I feel sorry for them) very carefully followed the spec, and the examples in the test schedule, and did this, like we did.

They will not work with almost any of the other CPs now live. They have to make major changes, now, when live. Really sorry for them.

Helping?

Seriously, we all need to work together. We have a test system that can help these new small CPs, and I am happy to help. https://notsco.co.uk/

One Touch Switching now Live

Today is the day!

There are now 166 providers on One Touch Switching. Congratulations to all of those CPs that have put in the work to make this happen.

These are retail provides of residential, fixed location, IAS (Internet Access Service) and NBICS (Number Based Interpersonal Communications Services, aka telephone service).

If you are trying to switch from a residential, fixed location, service and your retail provider is not on the list, please do let OFCOM know. I don't see Starlink, so that is the first complaint to OFCOM today.

I suspect OFCOM will take this pretty softly, given all of the challenges, so I expect more CPs to come on line soon.

Technically, I have seen no problems yet today, but I have reported one ramp up issue which was before OTS started, 21:24 last night. Even so, it looks serious, so it will be interesting to see how quickly it is fixed. I cannot provide any details because of the ramp up rules.

Stuff not happening yet...

Some of the faster switching of broadband service (next day even) is not happening yet - OFCOM have extended the 14 day (actually 10 working day) lead time on migrating services. However, I don't think there is a reason not to switch quickly if changing to a new provider - expect that the new provide probably has to put physical infrastructure in place.

The faster switching of telephone services is not happening yet - it seems this is not something that One Touch Switching does right away (yes, that fooled me too). A new process is underway way to do this, and gradually, over time, the new zero day process will come along. For now it is still 4 or 9 working days depending on the switch type. This also means it is still possible for a number port to fail, but using One Touch Switching should reduce that risk (as it checks postcode).

Switching 07 mobile numbers are not changing, that still uses a PAC (Porting Authorisation Code).

Business services are not covered - but it looks like many CPs are allowing switching of business services if you can match the details correctly as a residential match - we are. There are, of course, a lot of business only providers that are not on the One Touch Switching system at all, as they do not need to be.

Things break

None of us are perfect, though in all frankness I believe A&A are more ready than a lot of CPs, and we have been for some time. Again, I am not allowed to go in to any detail on this due to ramp up rules.

If you have any problems switching to us, please let the sales team know right away - we are monitoring the process closely and have means to raise issues with the other provider, if necessary.

If you have any problems switching away from us, you should normally contact the new provider, but you are welcome to contact us and we will be happy to look in to it. As I say, none of us are perfect.

Do remember our telephone services are not fixed location services (they are not a landline replacement) so are not covered by One Touch Switching. The non OTS porting process, as used for business services, will need to be used to switch number away from us.

Good luck to everyone involved. It may be a fun few days!

One Touch Switching

We have an interesting one today!

I have been reporting on the progress of One Touch Switching, and some of the many issues. To be clear, we are all working in the industry to make it work.

We are live on the "ramp up", and that has mostly gone well. We have some live switches going though. But there are also issues.

We have an interesting situation where, if, for example, we had a peering problem with some telco, or, well any problem with calls or internet or anything, with another telco, we would, reasonably, explain in a status post that there was an issue, who it was with, and what we, and they, are doing to resolve it. We are very used to that.

There is no point in saying "packets to *someone*, but we can't say who, are not routing correctly", or "calls to *someone*, but we can't say who, are not working right now". That would be utter nonsense.

However, I got a "slap on the wrist" for reporting on our status that one, or more, telcos was not handling match requests at present, were working on it (this is all a trial, really), and will be fixed soon. This is because I named them and explained publicly the problems they have.

So I am at a loss.

Do I remove them from the list of CPs on the ordering system? If I do I will be making it very public who they are by them not being on the list?

Do I say nothing, and leave customers, as one did today, trying over and over again to get a match request to work? But if I do not remove from the list, anyone trying a switch from them will see the problem - if I am not allowed to say there is a problem with them, do I have to somehow FAKE the system to pretend there is not a problem??!

I really cannot see the issue with saying what the issues are with the HUGE CAVEAT that this is ramp-up/trial. So once again a policy from TOTSCO that makes no sense.

But, I'll follow the rules. So sorry, some CPs are not working on the ramp up One Touch Switching, but you'll have to guess which they are. Good luck.