ESP32 Interrupts

Well, it has gone well... A week in Greece and a week back home, and I have my alarm system / access control code working.

In the process I have released an ESP32 / ESP-IDF based PN532 library which works with my DESFireAES library, and also a VL53L0X library.

The only bit that has proved problematic is the 9600 Baud RS485 Galaxy Bus library. It worked well on the ESP8266 using around 28800Hz timer interrupt to bit bash RS485. I coded the same on the ESP32, with much faster processor, and, well, two processors. It is being "iffy". It is close enough for now, but I need to fix.

I could go for using one of the three UARTs which includes some RS485 stuff, but that too needs some interrupts to get timing right, I expect. Getting the UART interrupts working was even more challenging than a simple timer interrupt.

I am working on updating my door controllers now. The new design is nice, and smaller than the old design.

So I'll read up more on interrupts :-)

ESP32 progress

I am, of course, reinventing wheels, yet again with my move to ESP32 for IoT and access control stuff.

It is going well - I am actually on holiday in Greece, but whilst my wife and daughter sit on sun beds, I have been messing with code. Yes, I took a laptop, programming cable, some ESP32 boards, and even a hand held oscilloscope on holiday with me. Sorry. I spent 4 hours on the plane reading the data sheet on my iPad!

My plan is to get my ESP32 development environment and toolkit working like I had on the ESP8266, and build up my "Solar System" alarm and door control module code.

In some ways this is a nice project - it is often good to rework and redesign something for the second time as you have learned lessons and can do things better. It is also nice that my code effectively has a specification to work to. The previous code was being specified, designed, and coded all at once. But now I have a system which works with the ESP8266 modules and should work just the same with the ESP32 modules.

Progress

I am making good progress. I have the build environment on Mac and linux working. The instructions were very clear. I am actually ssh'd to my linux box to develop now. I have the basic support system which allows me to connect to the WiFi, and MQTT server, and do over the air s/w updates, and store settings in non volatile storage. These are all based on the ESP-IDF tools, which seem quite good.

I have hit a few snags - for example I cannot do certificate pinning on the TLS using the SHA1 hash of the certificate, it seems. I have to include the PEM format certificate itself. This seems needlessly complicated, but not a big change.

I have a snag with MQTT that it seems not to pass on messages with zero length data for some reason. I may have to work that out and do a pull request to the library.

I also have to restructure the design in some case - for example, previously I did the settings in a block of memory that was flashed. Now I am using the provided NVS library for settings. It seems good as it does not have to re-flash a block on every change - it incrementally updates a log of settings in flash as it goes. Quite sensible really. However, it seems to lack a means to enumerate the settings that are there - I have to ask for a setting by name. As a result I have changed my code structure to register settings from the application with my library. This has meant a redesign, but as always it makes it much better in the long run.

Update: As redesigns go - I ended up re-doing it yet again, and pretty slick now.

I should be able to sort the basic GPIO (inputs and outputs) really easily, and hence use almost all of my existing door control code with that. It looks like I can compile my linux version of DESFire stuff for NFC too, so no need for an ESP specific version (if needs ESP specific AES calls).

There will be some challenges with the PN532 library and VL53L0X library I expect. But they are not rocket science, thankfully. The Honeywell Galaxy RS485 bus should just work if I can work out the timer interrupt logic. Saldy each of these is simple in theory but getting my head around the specific way of working for the ESP32 takes an unknown amount of time. Interrupts were a challenge on the ESP8266!

PCB design

This had me kicking myself the night before we left on holiday. I have made up several ESP32 based PCBs. A general break out board; a door controller board; and several Galaxy keypad boards. I had checked I could load code on the door controller. It worked.

But I realised, on reading more of the data sheet, that 6 of the pins on the bottom of the module (ESP32-WROOM-32) are connected to the internal flash and so unusable. Why even have them come out of the module then? They are literally in the way of tracking the usable pins out of the chip. I had the same issue on the ESP-12F and taped over the pads. The ESP-12S sensibly does not have them, but then has a GND pad you have to tape over if you have tracks under it. The ESP32-WROOM-32 has both unusable pins on the end and a GND pad in the way. Arrrg! Even worse, it is only 6 pins of 10 at the end that you cannot use, so not actually easy to tape over just the 6 and still sensibly solder to others. I had read a web site on "safe" GPIO pins, but that documented the ESP32 chip not the ESP32-WROOM-32 module, and for that the pins 6-11 are "safe" to use as GPIO. There are other pins that are iffy - i.e. needs to be high or low at boot. Those I knew about.

The end result is that the keypad boards I made would simply not work! So I was left with three boards in total to take on holiday. The door controller would not "work" for the GPIO pins, but they are not connected to anything that would stop the chip actually working. So usable for developing code. I have managed to trash one board already with a flash / secure boot mix up!

The first thing I did when I got here was re-work my designs on the basis of taping all 10 connectors on the end of the chip. If I had not, it would have been bugging me all holiday. I had to be a bit creative in some cases, but I managed to redesign both boards. I'll mill them when I get back (I don't think my wife would have let me bring the milling machine on holiday for some reason).

Door controller PCB

Update: The RS485 is a challenge. I made a low level bit bashed solution for ESP8266, which I could use. It uses two pins, a Tx/Rx and a DE/RE pin. But the UARTs in the ESP32 have hardware support for RS485 as do the drivers, and are better done with separate Tx, Rx, and DE (RE permanently enabled). So I may re-track the PCB and make a driver using the UART. Again, a redesign of both PCB and code. Overall this is dragging on longer than expected.

Security

Security with any IoT stuff is important, and reading the data sheets on the ESP32 and using the build environment, I can see that Espressif have done their homework. Nothing is 100% secure, but they are really trying hard here - which is good news as a lot of IoT stuff is totally crap!

The chip has a secure boot mode. This means the first stage bootloader in ROM checks the signature on the bootloader in flash before running it. The second stage bootloader checks the signature on the application. Doing an OTA (over the air) s/w update also checks the signature, even. This means you cannot load wrong code on the chip!

It also supports encrypted flash. This means that I can have the code, and all my settings (like login details to the MQTT server, and WiFi) securely stored in the flash. Even pulling it off the board won't get you the data.

It has a set of eFuses, which are built in to the chip. They can only be "blown", so are bits you cannot unset once set. These include bits that stop you reading the eFuses, but they are wired internally in the chip hardware allowing it to encrypt and decrypt the flash. You can even set a fuse to disable the JTAG debug port. This means you can really lock it down so that you cannot get credentials out of the chip or the flash. It might not stop someone de-capping the chip package and using a microscope to read the fuses, but it will stop most conceivable attacks.

They have built in AES, SHA, BIGNUM (RSA) and TRNG hardware as well, and the API makes using https and TLS simple and encouraged.

In short, security seems to have been done very well, and thought about at the outset in the hardware and software design. This is almost unheard of in the IoT world, which is generally pretty shit at security. Well done Espressif.

Update: I said nothing is 100% and a recent tweet suggests hackers have managed to crack it using clock or power glitching, but Espressif have replied saying they are adding hardware changes in the next release out soon! So they do seem to be trying, which is way better than most IoT stuff.

Opportunities

Once I have this starting point - code and PCBs - and it is working, then (apart from updating my existing door controllers), I have to think of other uses for these chips. They are packed with stuff - included a dedicated separate ultra low power processor. The opportunities for very low power devices are interesting. Even just the bluetooth looks fun - I could make it that door entry using a key fob needs to be able to see my phone in range, for example. I am sure there are a lot of opportunities!

They have some interesting direct AP-less WiFi modes in the API as well, and some low rate long range (e.g. 1km) WiFi functions too. There is even a MESH WiFi system. This all makes for some very interesting possibilities to make systems that do not rely on infrastructure such as access points.

I still have not made my environmental monitoring device yet, so that may well be the next project after this.

ESP-01, ESP-12F, ESP-12S, ESP32

As I mentioned, I am trying to move my IoT development stuff to ESP32, and notably move from Arduino IDE to the native Espressif ESP32-IDF using mostly C.

This poses a slight problem. The development environments are sufficiently different that it is something of a pain in the arse to maintain both. Probably just about possible, but not simple. So I'd like to basically abandon the ESP8266 / Arduino stuff (I'll leave on GitHub) and start working only on the ESP32 based stuff.

This poses a slight snag as one of the use cases for this IoT stuff is a very small device placed inside a Honywell Galaxy alarm system keypad.

This is based on an ESP-01 module which is ESP8266 based. It is not ideal as not CE marked, no RF can, very limited IO pins, but it is small. It comes with a 2x4 0.1" pitch header and is just the right size to fit a Pololu 3V3 regulator between it and the PCB, which is rather neat.

This use case has the ESP-01, a 3V3 regulator, an RS485 SO8 driver, and a small 2x2 milli-grid header for programming and debugging.

The whole thing fits in the small space inside the keypad, as per the picture on the right.

As a result I assumed I'd be stuck with, at the very least, my keypad code on the ESP8266 / Arduino based code.

The other IoT devices are generally based on either ESP-12F, or preferably ESP-12S modules. These are only slightly narrower than the ESP32-WROOM-32 modules I am now using, and so easy enough to make alternative designs using the ESP32.

The ESP32, however, whilst larger, has a lot of IO pins. It is possible to track these pins so that you have quite a few usable IO pins, and GND and 3V3, all on one end of the chip if you try :-

I have actually tracked Tx/Rx programming pins, and GND and 3V3 "through" some unused GPIO pins, which allows its all to come out of the bottom end and keep the whole thing quite narrow. Obviously this is fine as long as I don't make those pins an output :-)

Once again, the new milling machine is doing a great job, and I was able to solder it.

And, to my delight, it fits, and the keyboard PCB fits over it with no problems!

So, yes, I can actually move away from ESP8266 now. I have to migrate and re-work the various tools and modules I have made, but once done, I can continue development on ESP32 only. Yay!

P.S. Always tweaking :-)

Moving to ESP32

Someone already told me that once I move to ESP32 I won't want to go back to ESP8266.

Well, apart from the fact the ESP8266 is available in a tiny ESP-01 package, I think they may be right. The ESP32 in a WROOM-32 package is slightly wider than the ESP12 package, but because the GPIO pins can be mapped any way you want, pretty much, the tracking takes less space. So overall I was able to make an equivalent board in less space using ESP32.

The big challenge for me with ESP32 was the concern over the fine pitch - 0.9mm contacts at 1.27mm spacing. I was not sure I could either mill a PCB or solder a board at that pitch reliably.

I was wrong, I can!

New door controller using ESP32

Fixing the X axis backlash on my mill was a big help as well. So yes, I can use the ESP32 modules.

The other concern was the price, but it seems that getting a proper ESP32 is as cheap, if not cheaper, than an ESP12S. Yes, some cheap ESP12S can be got, without CE marks, and missing FCC ID, and not quite working well on WiFi. But £2.91 for ESP32-WROOM-32 from a reputable supplier is pretty good.

There are advantages to the ESP32, obviously, including much more RAM, faster, dual processor, more GPIO, more UARTs, and so on. It is a good comprehensive IoT device.

The other big change I want to make is moving away from Arduino IDE. Yes, Arduino makes things easy, but it turns out the Espressif IDF for ESP32 is pretty good. It has the key things like https, MQTT, OTA, and all sorts as standard.

So my plan is to make an ESP32 IDF based version of my door controller, including PN532, VL53L0X, etc. This should not be too hard, and a lot of my existing code can easily be ported to a simple C code environment.

I am, however, impressed with the documentation - not only of the APIs in the IDF, but also the technical reference manual. It actually details all of the registers clearly.

The other impressive thing is Espressif seem to to be trying to encourage some best practice in IoT. They have secure boot and signed images as a standard feature. Even https libraries, as standard. It means that making a reasonably secure IoT device is easy. Well done.

I managed to get the build environment working on Mac and linux with no problems, following the clear instructions. I built the "hello world" example, and flashed it with no problem.

I am off on a 4 hour flight shortly, so I have the 669 page PDF of the technical reference manual on an iPad for some light reading on the flight.

Of course, once I have this off the ground, I can look at Bluetooth, and CAN bus, and all sorts of new fun things.

New milling machine

I had a CNC 3018 PRO machine from Amazon.

• It needed assembling, but I've done that now. It is something to bear in mind if buying one though.
• It is a bit of a "home made" style, with the PCB bolted to the frame, etc.
• The one I got had a bent screw for X axis, which limited the usable space a bit.
• It has a working area 30x18cm.
• I was able connect a Z-axis probe (well, contact to PCB copper clad board).
• It works well, and I used bCNC on my Mac to work it.
• It comes with a controller box, which is not needed really!
• It is surprisingly cheap.

So, especially given the bent screw, I decided to get a replacement, and went for a CNC 3020 from Amazon.

• This is a much more "industrial" unit - proper drag chains and cables. Looks good but costs a bit more.
• Work area (you guessed it) 30x20cm
• Comes pre-built
• Has a boxed controller and power supply, with nice connectors to the machine.

However, it has a challenge or two - the main one is that it works with some very specific software (a pirate copy of which is apparently supplied with it). It has parallel (!) and USB, but the USB only works with that software and only on a windows machine - it does not even appear on a Mac even in a debug log! The software did not look too good anyway! This is a bit of a bugger.

The upgrade

My solution was to upgrade the controller to a TingG controller. This adds a bit to the cost (especially with the customs/VAT and admin charge from US). But was not actually at all hard to do.

The old controller is bolted to a heat sink, and can simply be unscrewed.

The new controller is slightly smaller. I needed to drill/tap an M3 hole in the heat sink for one of the screws so that I could screw it to the heat sink. I also cut an extra hole in the case for the USB connector.

Also, I decided to use the heat sink, which meant something between the new controller board and the heatsink. I got some small copper block / heat sinks and thermal tape - and I used them between the back of the new board and the existing heatsink.

The TinyG is designed to work without the need for a heatsink, but this can not really do any harm.

The connectors on the old board were not the same, though the new board has pads which means I could have changed them. I decided instead to use the screw terminals on the new board, fitting bootlace crimps on the wires. I had to swap the cables to make them fit, and use the motors 2, 3 and 4 as closest to edge of the case. But they all fit!

I also connected the emergency stop to the reset pins. I have not worked out how to connect the spindle drive to the new board yet - that is a challenge for another time. The case has a button and a dial for that which works for now.

I also got a 6 way (plus ground) chassis DIN connector and wired up all 6 end stops (X/Y/Z Min/Max).

This meant various micro switches, and super glue on the actual machine.

And also, the Z-Min I wired to croc clips to allow me to use contact to PCB for homing Z axis.

The TinyG has good documentation, covering setting the current, etc. I was able to set the axis to the right motors, and set the polarity and travel per rotation, etc.

However, the bCNC code I was using did not like the TinyG, and I ended up installing CNC.js which works nicely on my Mac. It does lack the multiple point Z axis levelling of bCNC sadly, but works.

I did have to configure the homing for Z axis a tad as it is designed for a switch from which you back off. For PCB contact you get the exact Z home, and do not need offsets. But this is all well documented.

Slack (backlash)

However, even though it is a really good solid construction and seems to have no play, I found there was an issue with the X axis. It seems to have around 0.1mm slack on it. This means if you move right to a point it is actually around 0.05mm short of it, same if moving left (i.e. is then 0.05mm right of it). Make a row of left and right moves and lines and you see the problem clearly. This test shows it well - centre is left and right moves before each line, but left is all moves from the right (apart from first, bottom left) and right is all moves from the left. Repeatable slack!

Oddly the Y axis is absolutely spot on!

This is not really enough of a problem to cause issues making PCBs, but is annoying so I wanted to fix it.

I checked for anything loose, and also checked the TinyG for any options, but to no avail.

My solution (now updated to the eps2gcode tools) is software to compensate for slack. This seems to work.

Overall I am quite happy with the result.

P.S. Tightening this nut a bit fixed it.



P.P.S. After reading the TinyG docs I am actually running at 4 micro steps not 8, so 1/200th mm spacing, which I think will be more than adequate for anything I am milling. I suspect 1/100th mm would be fine but actually that is a lot more noisy. Now that the backlash is sorted maybe I'll go back to 8.

Body cams are weird #GDPR

Body cams are weird - you walk in to a shop with one on, especially with a big WARNING: VIDEO/AUDIO RECORDING sign on it, and you get strange looks. But at the same time, half a dozen CCTV pointing at you, is "normal".

I have yet to be challenged on this, and I am not sure I really want the altercation, but I imagine if asked to "turn that off" it would be valid to say "I will, if you turn that off [point to CCTV]".

Let me stress, I am not a lawyer or GDPR expert, but I'll try and make sure this post is technically accurate and update any errors pointed out to me. And thank you Neil for the links, case references, and definitions and correction of my many typos.

Domestic purposes

One of the things that is outside the scope of GDPR is processing of personal data by a natural person in the course of a “purely personal or household activity”, which is commonly called processing “for purely domestic purposes" This is a tad complicated in interpretation though, and there is some confusion on this.

There has been a judgement on fixed position CCTV cameras recording in a continuous loop in people's homes (domiciles!) that says that it is only domestic purposes if there is no view of any area outside the property boundary. i.e. if the CTTV covers the road, or neighbour's property, it is not domestic purposes and hence you need a lawful basis, a privacy notice, to comply with data subject rights, and everything else that goes with GDPR compliance.

Personally (and I think I am not alone on this), I think this ruling is wrong: The cost seems to have confused the setting of the processing with the purpose of the processing. The classic example of domestic purposes, an "address book", will have names and addresses and phone numbers of people not within the boundary of my property. So clearly it is not the data subjects that define "domestic purposes" but the "purpose" of the processing (the clue is in the words used!). But as we have a ruling, that becomes risky to try and argue that CCTV covering the road is domestic purposes.

Indeed, for security of my home (domestic purposes) I want recordings of people "casing the place" (on the road or opposite my home) as that may have much better, daylight, views of faces to use when later the same vehicle is caught on camera relating to the place being burgled.

Dash cams, and helmet cams, were at that point considered "OK" though (or, at least, there has been no case law yet involving them), which leads to a whole series of scenarios where one cycles home and puts the helmet cam on the wall of the house, and then connects to power, and so on - at what point does it change from "OK" as a helmet cam covering the road, to not "OK" as a CCTV covering the road?

So, to be clear, if you have seen my body cam and found this post because the URL on the label, my purpose in recording is domestic purposes - I record my cycle rides mainly. Just like a dash cam in a car.

Continuous recording

There is some draft guidance on the matter of dash cams which actually suggests that dash cams must not  be continuously recording!

Yes, you read that right - it seems to suggest that you have to start recording just before the "accident" you want to record. I know, that is utter madness.

Of course there are plenty of people who will jump in and point out the technical solution is a pre-recording buffer. That is only saved if you "press the button" (after the accident), or the camera detects a collision, even, which some can!

But that *IS* processing personal information, even if it just goes in to RAM and is replaced after 5 minutes. An example of why that is processing is simple - imagine a dash cam where the camera owner is the cause of an accident. They have a choice about processing of that personal data in the RAM of their camera - they can choose to "press the button" and save the (incriminating) evidence, or not. The data subject can, if they are quick, ask for a copy of that data, at which point they have to "press the button" as otherwise they are deleting personal data after being given (what used to be called) a subject access request. Clearly the data in the pre-record buffer is personal data, and there are data processing decisions to be made in relation to that data.

We can only hope that such a nonsense paragraph is removed from this draft guidance (or, even better, replaced with something more akin to common sense, that someone trying to protect themselves from an accident, or put themselves in a good position if they do suffer an accident, is “domestic purposes”, even if it films someone else in a public setting).

Transition from domestic purposes to commercial

There is then one area about which I am really unsure. If I have personal data, recorded purely for domestic purposes, e.g. my address book, but I then tweet that!

In another recent case, the Court of Justice of the European Union has held that "since [someone] published the video in question on a video website on which users can send, watch and share videos, without restricting access to that video, thereby permitting access to personal data to an indefinite number of people, the processing of personal data at issue in the main proceedings does not come within the context of purely personal or household activities”.

Applying this to a dashcam or helmet cam, even if my filming would be considered to be “domestic purposes”, posting the resulting video on Twitter or Facebook (e.g. to highlight bad driving, or just as part of a journal of my day), unless I lock down access to the video, it looks like I would be outside the scope of domestic purposes.

Given how many people live out their lives online, engaging in the type of chat that one might previously have had in a pub or coffee shop, this feels like it could entail the imposition of GDPR — which, let’s face it, is not something normal people should have to understand — on the general public, for very common activities. It seems to me like rather too much of an interference with people's private lives.

Somewhere in between

Sharing a family video (e.g. my grandson riding a bike for the first time) with family is clearly domestic purpose.

I assume doing so over iMessage is still so, as I do not hand the video to Apple (end to end encryption).

What of posting to a small (family) group on Facebook? My "purpose" is clearly domestic, but Facebook have the image and they do things with images that are not domestic purposes.

What does that mean for GDPR I wonder? I assume that this means that Facebook needs to comply with GDPR, but I do not (since I am not sharing with an indefinite number of people).