FB9000

I know techies follow this, so I thought it was worth posting and explaining...

The FB9000 is the latest FireBrick. It is the "ISP" high end model we do. We do smaller models like the FB2900 as well, but FB6000 and now FB9000 are aimed at ISPs and the like. It is what A&A use.

You can see a lot more here: https://www.firebrick.co.uk/fb9000/

But why now - the FB9000 has actually been around a while?

We are not like other companies!

When we launched the FB9000, we obviously started using them ourselves, in A&A.

We hit some snags, some random crashes, we backed off, we found a release of the code that worked and was stable, but that does not address the underlying cause. Why did some releases crash? So we were able to continue with a good set of working LNSs on a somewhat aging reliable release of code. But it meant some inconvenience for our customers along the way when we tried other code. We do not like that! So we massively backed off.

Thankfully some devices, notably BGP routers with VRRP, which annoyingly crashed far less often, can recover in literally 1/10 of a second. So they were good test cases for new code without upsetting customers. An LNS does not recover as well as all users need to reconnect and that can take minutes, depending on their router.

You would not believe the details behind the problems, seriously, it is crazy, and I am not even going to try to explain it here. There may be a really detailed technical blog post by the FireBrick team in time. Suffice to say this snag held us back something like a year.

Now, we could have plowed ahead, and sold loads, but we were really careful not to. A couple of ISPs trust us enough to solve it that they have the stable code release running and did buy some. Thank you. They did so very aware of the issues and have been fine on the stable code release.

It takes time

The issue is that the fix literally takes months to be sure it is a fix. And at A&A we have been doing very very careful staged upgrades to LNSs to prove this, with a lot of staff working during the night to manage this (well mostly one, thanks Andrew). This has taken months even after we think we have nailed the underlying issue. Thank you to all of the staff involved.

We are now at the stage we can probably say it really is fixed, at last. But it is one of those things which are a problem - you cannot be 100% sure until it doesn't crash. Yeah, when exactly is that?

Chasing ghosts

We really are pretty damn confident now. The issue is that, as an engineer, you want to find the smoking gun. This issue is a horrid mix of hardware quirks that even the chip manufacturers cannot explain, and some very very subtle hardware initialisation that has impacts days, weeks, even months later in running code. We have found some concrete issues, well, things not quite 100% as they should be, but not the causal link you want between such things and the problems we saw. And this is not for a lack of trying - every time we thought we found the cause the team have tried hard to break it in a repeatable way. To overdo what we may possibly have done wrong.

A product we can sell

This has always been an awesome product, and any other manufacturer would have fired off the marketing team years ago for sell - sell - sell.

We finally have something we can say with a lot of confidence works well. Does the job, and does it well.

There is more

The FB9000 is awesome, and if you are an ISP you really want one - they have some unique features that really gives A&A an edge which you too could enjoy.

But we are working on a next generation for the smaller units, the FB3100 to succeed the FB2900. It too will take time, and we hope none of the same issues. The FB2900 is also awesome, and there are some offers I think on the pricing soon.

Audio sync

Should audio sync be an issue?

Well I would hope not.

The way I do it is that I record video with my nice Canon 1DXIII which has audio, and audio with an audio recorder on my shirt, and tell final cut pro to make a synchronised clip. Simples.

The camera audio is not good, it ends up with mechanical noise from lens focus, and echo, but the audio record on a shirt mic works well.

I have since changed from using a Tascam and radio mics to my own audio recorders for my shirt.

But someone asked how to sync audio with my new recorders?

I was "well, sync to the (crap) audio on the camera", which fooled them as they were talking of sync of raw video only and raws audio only.

So I made a feature. My audio recorders will now make a clear 1s tone at start of audio and exactly aligned a clear red LED on the front.

This makes aligning video and audio easy.

But I never expected to use it. Well, until I did.

I recorded a video, direct on final cut pro, on my mac studio with monitors, and weirdly they have bad audio sync!

P.S. someone said would be nice of LEDs off during record, and now that is the case for dark mode.

ESP32 SD Card wiring and driving

This is so that people like me find it and know what I learned.

Making an ESP32 board with Micro SD slot, in my case specifically an ESP32-S3

First step, wiring up. This is what I have ended up with.

There were other stages, including just the pins you need for SPI mode, which mean no SDDAT1 or SDDAT2 lines. But with them having a pull up anyway. For SPI they have different meanings and labels, typically. The other stage was no resistors.

Resistors or not?

This is complicated, honest. After lots of reading of all sorts of pages and comments it seems...

• The resistors are part of the spec, so highly recommended.
• The resistors are needed for the default, old school, initial (slow) mode by which SD cards work as they use open drain driving (pulling down only) and need resistors to return signal back up. But, you don't stay in that mode - you switch to a mode where pins are driven both ways, so why does it matter?
• Some cards even have internal resistors, it seems. But no ideal which. So some may "just work" anyway.
• The ESP32 has an option of pull up resistors, but these take time to rise, and so do not work well if you have no resistors - the practical impact is really slow data transfer (using SPI mode) - but otherwise working. You can actually work around this with driving the CS in SPI mode low rather that using the library, it seems, at least for SPI mode working.
• The resistors also help avoid odd power usage on floating pins even if not being used, but again, you change mode to one that does not use open drain.

End result, just simpler to have the resistors, even if that means squeezing them in on a small PCB.

GPIOs

This is 7 GPIOs, a lot, but they can be mapped to adjacent pins, and the right order, to connect directly to the Micro SD card slot. SPI mode could use only 5 if needed, but best to have all the pull ups anyway.

SPI or MMC control libraries

Lots of examples of SPI wiring and SPI drivers, leaving some NC pins or leaving them with just pull ups. This is what I started with. But it was not as fast as I wanted (reading and writing around 250kB/sec).

It seems the MMC drivers are way faster, simpler as that, even when using 1 bit. I now have boards coming with 4 bit working to compare. With one bit I am writing 350kB/s and reading 750kB/sec, so may be the specific SD card is slow - more tests needed, and will be interesting to see 4 bit mode -  I'll update this post.

Small LED strip boards.

I have these. 

The idea is small, really small,. LED driver, ESP32-S3, but small. Power from strip 5V-35V, so a switch mode supply.

To do that no USB, not USB-A or USB-C. I have used TC2030 pads.

But to allow those wanting to solder manually, pads for that as well.

We are talking 27mm x 16m LED controller.

Fun with... LED Strips

I am not late to the party here, honest. I have been playing with these for some time, and even design and make LED controllers.

The ubiquitous protocol is "WS2812", which is basically a serial data stream on one data line at a rate of around 1MHz with short or long pulses being 0 and 1 as data, and a long gap being end of sequence. The exact timing in the various data sheets do seem to vary slightly. SK6812 is similar.

The data sequence is binary values, 8 bits per byte, and the bytes that are each colour. The exact colour order varies depending on chipset it seems (patent/copyright/god knows why). In general it is Green, Red, Blue, but can be Red, Green, Blue, and I have seen others.

Each device consumes its data sequence for its colours, and passing on the remaining stream. Once the long gap comes all devices load the accepted colours to their LEDs simultaneously, which is cool.

There are also RGBW, which have four bytes not three, so include a "white" level. This is cool (or warm) as the white LED is often larger, and brighter, and the power of one LED but white (doing white with R+G+B is not as nice a white, and uses 3 times the power).

Coding for this is simple, and ESP has ways to do it - typically using the "remote control" system for IR remotes, but can also be done using SPI bus with a little fun with the logic. This allows working with many devices as a serial stream via SPI is very simple.

They also get smarter with some designed to take a backup data which is a connection of the data in to the previous pixel. If the backup comes in without the expected data, it can accept the backup, one pixel later, as its own data, and pass on as such. As such a single dead module does not break the chain.

So, we have many choices, the exact chipset, the order of colours, if white included (which flavour of white such as cool, warm, natural, etc), if backup line, and voltage.

The voltage is a fun one - commonly 5V, the voltage drop over a long strip is too much and you lose blue, then green, so have to inject 5V at multiple points in some cases. But 12V has options - one is 12V drives three sets of RGB LEDs in a chain, so 3 "pixels" the same, but another is 12V works one "pixel" (the WS2815 range do this). I have, however, some that work on 3.3V (impressive for blue) for on board LEDs, and I use these all the time on my PCBs - they would not work on a "chain" of LEDs off the board.

What would be nice, and I am not sure I am quite seeing yet (please say if not), and seems the logical conclusion here would be.

• 12V supply (working all the way down to 5V or less)
• Per pixel control (not sets of 3 pixels together)
• Backup data line
• RGBW option so white on the pixel (in choice of colour temperature)

It looks like the last point is the sticking point from what I read, as WS2815 does the rest, but when finally common place, that will be the ideal solution. The 12V working allows long strips on single supply. The W allows lower power and brighter "white" as functional lighting. The per pixel control is obviously better. The backup is obviously prudent. It would be extra cool if it can run from 5V on the same chipset, just obviously suffering the voltage drop issues over a long run.

For extra fun, I tested the WS2815 (12V) modules and they work very happily from 5V, so allow a hell of a lot of voltage drop. The data sheets say +9~+13V

I²S

I²S is, err, fun.

What is I²S

Well, first off, it is grammatically like I²C which is an acronym with two Is in it which people then treat an acronym like a mathematical equation and so make it I²C. I²C is IIC. I²S is the same annoying logic, it is in fact IIS, which is Inter-Integrated Circuit Sound, which wikipedia says is pronounced "eye-squared-ess"[citation needed]), and yes, I know nobody that says squared! It is eye-two-ess in my book.

But what is it exactly?

Well, it is a standard (and I use the term loosely) for audio over digital signalling.

What this means in practice is digital microphones and digital speaker driver chips. And to be honest I am amazed. This is clever tech, and pretty much a result of mobile phones.

The options

One challenge is that there are many options. It seems, at least, you may need.

• MCLK which seems to be a master clock at a higher frequency. Now, whilst the code and hardware I use in ESP32 understands this, it seems it is usually not needed.
• A WS or LR clock (I assume WS is "Which Side", and LR is "Left/Right").
• A BLCK - a bit clock.
• A data line for the clocked data in or out.

This is not too bad, especially if MCLK is not needed.

Basically the WS clock is slower, and the BCLK is faster and it allows for each side to have a number of bits of data to be clocked. Simple enough.

PDM mode

The first thing that threw me is that there are I2S microphones that use PDM. I really don't quite grasp the logic here, sorry. PDM looks simple enough (mono) as it looks like you have on/off period that relate to the level of audio. But I am uncertain how that works as clocked left and right.

The PDM microphone I tried allows up to 4.8MHz clocking which is way over audio, so clearly means clocking more and sampling that to decode the PDM.

Seriously I don't (yet) understand, but it works, and the ESP32 can handle it and get 16 bits per sample for each side at various rates.

Standard mode

This makes more sense - you have a BCLK that, say, clocks 32 bits and then every 32 bits the WS clock toggles. So the LR clock is the sample rate, and one side clocked on BCLK when WS is low, and the other when WS is high. Clocked MSB first, signed.

What is actually cunning here is how many bits per sample. A microphone could supply 8, 16, 24, 32, whatever bits, MSB first, on the change of WS and clocked each BCLK. Then stop, and that would be noise for any extra bits. So if you clocked something that only does 24 bits at 32 BCLK per WS cycle, you get 32 bit data where top 24 bits is meaningful.

Philips mode?

There is, of course, a catch! There is a Philips mode, which means the data clocked on BCLK is one clock later than the change of WS clock. But standard mode has no such delay! Oddly, it seems Philips mode is more standard.

Stereo or mono

The underlying format is always Stereo (well, ish), but the hardware on the ESP32 is not daft. I can say I only want left or right channel mono from the stream. Output is always stereo, and there is an option to say mono which I assume (hope) sends same data on each side.

TDM mode

There is, it seems, another mode, where WS is a short pulse at start of frame, and BCLK allows lots of channels, well 8 channels I think, to be clocked. Is this a hark back to 8 track tape?

Hardware

The ESP32 handles all these modes, yay!.

For stereo input you wire two microphones on the same bus, one set as left and one as right.

For stereo output the same, a speaker driver wired as left, or right. It can also be wired as left+right even.

What is amazing is the chips that now exist.

Microphone

TDK do tiny microphones, a PDM one, and a 24 bit per sample I2S one. They are unbelievably good.

Speaker

This was even more impressive - Maxim do a really tiny speaker driver, and it is 1.3mm square BGA that does it all - a cap on power maybe, but it takes BCLK, LRCLK, DATA, and drives a 4 ohm speaker, and that is it! Use two and you have stereo.

Why is this so good?

The simple answer is the hardware in chips like the ESP32 mean that audio in, or out, is a DMA behind the scenes process allowing blocks of data to be sent or received reliably by the hardware.

Before this you would need a good quality ADC sampled at a consistent high speed rate. You would need a good quality DAC updated at a consistent high speed rate. This was hard work. A chip for each was complicated.

Now the microphone is one chip, and the speaker driver is one very tiny chip. And that is it!

Fencing

Bit of fun...

We usually put up some Christmas lights on the house - some fairy lights on the metal fencing at the front, but a pain as means a cable out of a window. They are usually just normal fairy lights.

But with my new found expertise in WS2812 style LED strips, and my controllers, I decided to do better.

11m of wooden fence at the front of the house on the road. So let's do this properly. The key point is I have outside power at the end of the fence for the hot tub. So I was able to install, under cover, a 20A 5V power supply.

I then got 4 strings of fairy light style water proof 5V WS2812 LEDs.

I drilled nearly 200 holes, carefully measuring each to be level and evenly spaced. That is surprisingly hard work, LOL. James followed me poking LEDs through the holes. We were both expecting to fall off the damn wall, and James's main concern is I would fall off whilst he was not videoing!

But it is not quite so simple. Just in case you don't know, there are two common issues with LED strips.

Current limit

One issue is max current draw can be too much for power supply. To test you can either work it out, or, simply set all LEDs full white. 200 LEDs is too much for a typical small 5V USB charger plug. Hence the 20A 5V supply.

I actually also did 663 (11m) RGBW LEDs on nice 45 degree extruded trunking with diffusers for the hot tub as well, from same supply. Now that used a lot of current - just one 5m strip is too much for a USB 5V charger when white.

Voltage drop

This is slightly harder to solve. Along the strip the current draw means voltage drops as you go along. Different LEDs need different voltages. First you lose some blue making it yellow, and then some green, making red/pink. And even before that, when white still, you lose some brightness.

So with this 50 LED strip - one strip works. Two strips work but losing brightness at end. Three strips means going distinctly yellow at the end. I wanted four strips!

The solution

The solution is power feed in - the strips even have extra tails for power as well as the three wires for power and data. You feed in extra power at each strip end, so for my 4 strips I feed in at 5 points.

But how do you feed in power? In some cases you could simply power your longer strip at both ends and not have enough drop to the middle to notice. But I don't have power at the other end.

But actually it is possible to feed in even with just power from one end. The reason is the resistance of the wires, these are classic Chinesium™thin wire. If you actually have some thick good quality copper wire you can run and extra power lead the whole length and feed in at each strip end. This is what is in the WAGO boxes in the image.

Merry New year!

P.S. my son sells the controllers and stuff, https://hiwtsi.uk/

Update: Measuring resistance on the 50 LED strip power lines showed 1Ω but the leads were 0.1Ω, so 0.9Ω. A similar length of copper wire registered 0.4Ω, so 0.3Ω, so ⅓ of the resistance.

James did a video :-)