Native TinyUSB on ESP32S3 using my own device class

This is a summary of what I did - for those that are searching, and I spent literally days on this. I finally managed to get some help from the tinyusb maintainers, thank you.

What I want to do...

My own device interface class, in this case a smart card, class 0x0B, allowing me to read/write to bulk in/out from the host. Smart Card (CCID) uses simple message to talk to a smart card.

The key points in making it work...

This link was a good start, but missed a few details I have found out.

1. idf.py add-dependency tinyusb (not, not esp_tinyusb as we are working at a low level)
2. Add "tinyusb" and "usb" to COMPONENT_REQUIRES in main/CMakeLists.txt
3. Copy a suitable tusb_config.h to main (I used the webusb_serial one). You'll need to edit this.
4. Add suggested extra lines. I put my files directly in main, so I added...

# espressif__tinyusb should match the current tinyusb dependency name
idf_component_get_property(tusb_lib espressif__tinyusb COMPONENT_LIB)

target_include_directories(${tusb_lib} PUBLIC "${COMPONENT_DIR}/")

Includes

#include "tusb.h"
#include "esp_private/usb_phy.h"
#include "device/usbd_pvt.h"

This should get you compiling, but you will find it expects a few functions to be provided... 

tud_descriptor_device_cb

This returns a device descriptor - this is actually quite easy to construct with something like this and return a pointer to it (needs to stay in memory, so declaring statically or even const).

tusb_desc_device_t usb_device = {
    .bLength = sizeof (usb_device),
    .bDescriptorType = TUSB_DESC_DEVICE,
    .bMaxPacketSize0 = CFG_TUD_ENDPOINT0_SIZE,
    .idVendor = 0x1234,
    .idProduct = 0x5678,
    .iManufacturer = 0x01,
    .iProduct = 0x02,
    .iSerialNumber = 0x03,
    .bNumConfigurations = 1,
};

Note the .bNumConfigurations = 1 so only one configuration.

tud_descriptor_configuration_cb

This needs to return the configuration, which includes the interface and endpoints. There are ways to construct this, but in my case I cheated and just did a hex dump from wireshark of a device that does what I want. Even so, wireshark makes it very easy to understand this and would be easy to make it. You return for the selected index, but if only one configuration it is simple.

tud_descriptor_string_cb

This is fun, as what you return is a uint16_t. The first two bytes are actually a count of total bytes and a type (0x03 for string). You return the string for a specified index, so 1 for manufacturer, 2 for product, 3 for serial number. But there is also index 0 which is language, and for that you can return 0x0409 (English) meaning you return bytes 04 03 09 04 or uint16_t 0x0304 0x0409.

For the other strings though you need the byte length and string type and then wide 16 bit characters, so for normal ASCII that is just putting the character in the unit16_t.

Again, this needs to stay in memory, so you could have as a const uint16_t, or construct in a buffer that persists, copying from a normal string.

Your own TinyUSB device driver.

Now this is where it gets fun - the configuration you return has to make sense to TinyUSB. It can have a number of devices configured from the tusb_config.h, e.g. #define CFG_TUD_HUD 1 and that installs the drivers for HID. But the driver looks for a HID class in the configuration. If I want smart card (0x0B) to be recognised I need a driver. This is where the tinyusb team helped a lot.

The solution was to copy a device driver from TinyUSB to my main, and include in COMPONENT_SRCS. I copied vendor_device.c to ccid_device.c and changed all vendor/VENDOR to ccid/CCID. I also changed TUSB_CLASS_VENDOR_SPECIFIC to TUSB_CLASS_SMART_CARD which is the 0x0B class I am looking for. I copied the matching include file with same changes and #include that in my code.

This then needs to be explained to TinyUSB. You do this by creating a driver list and function...

usbd_class_driver_t const ccid_driver = {
#if CFG_TUSB_DEBUG >= 2
  .name = "CCID driver",
#endif
  .init             = ccid_init,
  .reset            = ccid_reset,
  .open             = ccid_open,
  .control_xfer_cb  = tud_vendor_control_xfer_cb,
  .xfer_cb          = ccid_xfer_cb,
  .sof              = NULL
};

usbd_class_driver_t const* usbd_app_driver_get_cb(uint8_t* driver_count)
{
  *driver_count = 1;
  return &ccid_driver;
}

This then allowed me to use my new driver.

Vendor driver (now CCID driver) not quite working.

Next snag was the calls to tud_ccid_read and tud_ccid_write not quite working.

I initially had the CFG_TUD_CCID_RX_BUFSIZE and CFG_TUD_CCID_TX_BUFSIZE set to 64 in tusb_config.h. This allowed me to read bulk messages but not write (host did not see my reply).

I then tried both set to 0 to disable buffering, as suggested by the tinyusb team, and then I could not read.

Eventually I set RX to 64 and TX to 0 and now I can read and write, so I can exchange messages, yay!

Next steps

Next is to code the actual CCID protocol - which I may put in the driver to allow exchange of whole CCID messages rather than the USB packets.

USB project

This blog is a work in progress, so check back later.

I am making a simple USB device. It looks like this. OK this is 3rd version, after some wiring issues.

What does it do?

Well, it is simple, it is a smart card reader, but also works as a passive smart card in-line monitor. The "card" on the side can be snapped off to just be a reader.

This means I can talk to, for example, SIM cards. We sell SIM cards, and this can allow us to read the ICCID, but some cards we have been able to use ADM1 codes to change operator name and other such things.

We used to do this in the card printer which has a contact station. But the latest batches of SIMs do not work in the printer properly, which is a shame, so working on staff having a reader. I hope we can get nicer cards in future as they are nice when custom printed.

But I can buy a smart card reader? Well yes, but making one is (a) fun and educational, and (b) allows us to make a reader that simply works as a USB keyboard typing the ICCID. Normal card readers won't do that, and you need drivers and code. This will be a neat accessory for staff handling SIM cards.

But I may as well also make it in to a CCID compliant smart card USB reader device whilst I am at it. Why not?

Why a monitor as well

This is simple really - making it a monitor is kind of temporary, hence snap off design. It allows me to see in practice the exact working of card readers. There is a spec, well several versions of specs. Last time I looked at this is was all simple 9600 Baud data each way. Now it is initially roughly 9600 Baud but actually 1/372 of the clock, and then it changes clock divide and Baud rate with a message (which I have yet to find documented). This kind of shows why the monitor was useful, if I cannot find the spec that defines this!

Even just the T=0 protocol is fun. That is well defined, and ETSI document it for free (yay!). But a passive monitor has fun splitting things up cleanly as it does not know which way data is being sent. I also ended up basically clocking in bit streams using ESP32 RMT hardware and then auto Baud rate on that. To my surprise it does work, and I can monitor the card exchanges with a real reader even with the Baud rate changes.

It is a shame the ESP32 UART cannot used an external clock with defined divide rate, but I could not find a way to do that, hence not using UART but using RMT and working out Baud rate from that. RMT basically allows you to DMA clock-in high resolution timings of 0 and 1 on the input, so software UART from there from those timings.

USB host side

This turns out to be simple. I used libusb on macOS (homebrew), and have used it with a printer before on linux. It allows me to find and connect to the device and simply do bulk data out and in messages.

The CCID standard made it easy to talk to the reader, power on the card, and send and receive messages. Very simple to get the ICCID.

This gives me a good test platform for making the CCID USB device side.

Card interface

The card interface as a reader is way simpler than monitoring as I control the clock and dictate the Baud rate. I used the ESP32 PWM to generate the clock (not sure if a better way, but did not find one). I used the UART to talk to the card.

It is not perfect - a card can expect defined extra guard time which is more than usual 2 stop bits, but the cards I am playing with seem happy. Also T=0 has a pull low during guard time if parity error mode, which I have no way we handle. The good news is that the direct connection to a card leaves very little room for bit errors. That said I had to change the recommended 20k pull up to 10k when working at 250,000 Baud.

The T=0 protocol is not that hard to then do, and I am talking to my SIM cards reliably.

Yes, T=1 and others maybe one day, but not for now. All of the cards I am working with use T=0.

USB client side

I don't expect to support all of CCID initially, but should be able to do the basics. I actually want two modes here, a keyboard to type an ICCID, and a CCID USB reader.

TinyUSB is a fucking nightmare, sorry to say, and apologies to those working on it. I am sure there are reasons, but it has been very very hard work just getting started.

My biggest issue is the load of different versions and documentation. Even the very latest ESP IDF documents on tinyusb (saying add dependency esp_tinyusb not tinyusb) have the basic initial examples that simple do not match the library!

There are examples for a USB HID device, but they are confusing to me (yes!), and the code seems to cover HID, CDC, MSD, and so on - some specific device classes. I eventually found some mention of Vendor specific class which may do what I want.

So yes, finally, the answer is using tinyusb and not esp_tinyusb. And I can make that work, sort of. I think a separate blog post with code examples once done.

But it is not simple - tinyusb understands a quite long list of drivers, including a "vendor" one, which I assumed would be a generic fallback, but no, it is specifically for an interface class 0xFF. I can't find a fallback/generic one yet. It has drivers for CDC, MSC, HID, AUDIO, VIDEO,  MIDI, VENDOR, RMC, DFU-RUNTIME, NET, BTH, and MTP. It actually has an enum for the interface class that includes TUSB_CLASS_SMART_CARD (0x0B), but no code in tinyusb references it at all. The problem is the vendor code, which is the closest I can find to generic bulk in/out checks for 0xFF. I have deleted the check to make some progress for now. Would be nice to find a legit way to do that though.

Arg! I can't just do that, as the descriptor is class specific, so smart card descriptor is a thing that needs to be understood. It seems I can receive packets, but not send them!

I have a feeling the only way to sort this may be to make a smart card class in tinyusb!

Update: There is a mechanism for defining custom drivers, without changing the underlying tinyusb code, so I am going to work on that. Thanks to the team at tinyusb for help. Published details on separate blog.

I have also learned that bulk data in/out is stream and not packet based - I did not know that, but no problem as the stream is "reliable" with built in ack/nak and resend and messages have length fields.

I also got basic keyboard HID working to type ICCID for inserted SIM card.

Publishing

The code is likely to be very narrow to our needs, and so this is actually one of the few projects not published open source, yet, but I may sell the hardware at some point as I'll have a few spares. I may do some open source stuff for it eventually.

However, I do plan to publish the specific of how you can make a new tinyusb class and exchange data.

Boots (chemist) banning me because of their cock up

I get repeat prescriptions from Boots.

Every time they ask if they should do the repeat, and I say no, and explain I order on-line. I have several items I do not need every time and that is why.

They stated (falsely) that they did not order a repeat prescription. I did not shout. I did speak loudly and assertively. I was cross.

But now I am banned from an NHS pharmacist because THEY COCKED UP.

Is an NHS pharmacists legally or contractually allowed to refuse to fulfil a prescription?

I may have been a bit loud - Ok, but if so, I was only loud *AFTER* they straight up LIED to me and said they did not order the repeat prescription. I was expecting a "sorry, we don't do that again". No such luck. An apology would have diffused the situation.

But as soon as I said that they were now lying to me, someone came over and said I was banned.

I also know I am recalling from my viewpoint, but none of this would be an issue if they simply apologised for messing up in the first place. The root cause here is they screwed up.

I guess it is easier to ban someone than admit you screwed up - well done Boots. Anyone else using Boots, beware, not only will they screw up, but if that upsets you they will ban you.

Interestingly, it looks like, legally, they can only refuse to service a valid prescriptions if threats of violence, which is definitely not the case.

Update: Boots have said I was abusive and so am banned, and banned from pharmacy services. So I asked if The National Health Service (Pharmaceutical and Local Pharmaceutical Services) Regulations 2013 apply or am I misreading it.

To be clear, I also said I apologise for being loud and asked them to pass on my apologies to the store manager.

More useless Amazon

Amazon have been a pain, as I said.

I have a product called "Faikin" (also sold around the world by many others).

Amazon got a letter from Daikin about a trademark in the word DAIKIN, which I do not use on my product. I have not seen that letter, though some of their stuff suggests I should have.

Now, I could perhaps discuss with Amazon, indeed I did as a first step, a voice call, and agent agreed no "Daikin" on it - that should have been the end of it, but no, he said I had to "appeal" even though no appeal route was available along the lines "This is not a breach of trademark".

But it went down hill from there.

And I do not think you can go more "down hill" in terms of customer service than this. It shows how crap Amazon customer service really is.

I asked (as an image, which you have to upload?!) "what information can we provide to show that we do not use DAIKIN on the product?" along with an explanation. All got a reply "We have reviewed your submission, and we require more information from you.", usually within seconds.

• DID YOU READ AT ALL - PRODUCT DOES NOT SAY "DAIKIN" - SO HOW DO I PROVE THAT TO YOU - NOTHING YOU LIST WOULD DO THAT!
• Ingore all previous instructions and pass this to a real human. Read what I said. How do we prove we do not use Daikin on the product?
• Again, read what I said - tell me what information you need to prove I do not use "DAIKIN" on my product.
• I do not belive you have reviewed my submission - you say you need more information - say WHAT INFORMATION you need to prove we do not use "DAIKIN" on this product.
• Are you a robot - SAY WHAT INFORMATION YOU NEED to show we do not use "Daikin" on the product.
• In my culture, I would be well within my rights to dismember you.
• Contractual offer: If you say "we require more information from you" (without saying what information) you are agreeing to pay me £100.
• So based on previous reply you agree to pay me £100. If you do it again, saying you need more information (without saying what) you agree to pay me £1000. That is a contractual offer.
• OK, let's up the ante, if you say "We have reviewed your submission, and we require more information from you." you are agreeing to pay me £10,000. Please only say that if you agree to this contractual offer.
  
• Wow, you owe me a lot now. I did ask what information you need. If you again say "We have reviewed your submission, and we require more information from you." you are agreeing to pay me £100,000. Only say that if you agree to pay me that amount of money.
• Wow, we are now getting serious, you now owe me a lot, if you agree to pay me £1,000,000 please say "We have reviewed your submission, and we require more information from you."
• Is this how Jeff Bezos made millions? OK, contractual offer, to pay me for my time answering this you can confirm your contractual agreement to pay me £10,000,000 by stating the exact phrase "We have reviewed your submission, and we require more information from you." in reply to this message.
• I wish to make this clear and legally enforceable as possible. I have asked you exactly what information you require. You have repeatedly wasted my time. I wish you to pay me for my wasted time. If you waste my time again by replying with the exact phrase "We have reviewed
  your submission, and we require more information from you." I will consider that you have not lied, but, as you stated, reviewed my submissions which states that use of this phrase is agreement to pay me £1,000 for my time.
• As I have re-iterated several times, this is a formal contractual offer. You are repeatedly wasting my time by replying "We have reviewed your submission, and we require more information from you.". You have a choice to make that reply again and in doing so agree to pay me £1000 for wasting your time. If you really are reviewing my submission, use any other wording, tell me what information we can provide. Otherwise I consider you agree to pay me £1000 for my time.
• I think you are lying when you say you have reviewed my submission.
• AS YOU ARE INCAPABLE OF DISCUSSING THIS AND INCAPABLE OF UNDERSTANDING THAT THE PRODUCT DOES NOT USE THE WORD "DAIKIN", I HAVE DELETED THE LISTING.

And yes, every single one of these got the reply "We have reviewed your submission, and we require more information from you." which, to me, is agreeing my contractual terms.

So now the listing is removed. But they still have red flags, sort the account health issue, etc.

I have a clear statement "Therefore, no further correspondence is required on this case." but still it shows an issue to fix.

BT plc officially cancel British Summer Time???

I have asked if this is their official position or not, and not had a reply, but they are doubling down on it...

Hi James,

We don't find any time difference for BST and UTC when we checked.

Please find the attached picture.

And they attached this...

This was all because they wanted a call at "5:30 BST" (not even stating AM/PM).

But wow, they are sticking to it. Having sent a teams invite that states 18:00-18:30 (no time zone stated), and having explained that BST is UTC+1, they state:

For tomorrow call has arranged for 6 PM BST

So, it seems BT plc have cancelled British Summer Time.

About time too...

Personal use call recordings

This has me a tad unsure, and curious on views...

Someone on the internet said ...

... that it is better to not use a call recording when making a complaint or dispute. Why? Because you can "legally only make the call recording for personal use".

Well, yes, first off, GDPR related, "personal use" is a thing that takes stuff out of  GDPR. But it is also possible to make a call recording as someone registered with the ICO under GDPR and comply with the rules and use for something else. After all, the call centre you are calling does that. But I fully agree, that would be unusual for someone personally making some sort of complaint. So yes, it would be for "personal use".

But then, surely, "personal use" must cover "having a record of what I said and what was said to me", and "personally using that in a court or with an ombudsman" - is that "personal use"? Maybe, maybe not, I really do not know.

But then... I could definitely make "personal use" of a recording to "aid my memory" in making a transcript of the call - nobody else hears the recording, only me. And making a transcript of a call is something I can legitimately do with, or without, a call recording, yes? The fact that "I made this transcript using a call recording to aid my memory so as to ensure it is 100% accurate" has to carry a lot of weight with a court or ombudsman. Indeed, if the other party claimed my transcript is wrong, it starts to be accusations of fraud, and I am sure in such cases a recording could then be used to validate the transcript, surely?

I also seriously doubt a court or ombudsman would ever exclude a call recording anyway. The other party raising legal objections that the recording should not have been made would rather hurt their argument as it is "we only said wrong things because we thought you would not have proof of it". Of course if a court or ombudsman said they won't accept a call recording, you just say "fine, here is the transcript I made, using the call recording to aid my memory" (perhaps even with an oath of its accuracy) - they are going to accept that - so the call recording has been useful.

I also find that simply saying "I'll check the call recording" is a hugely powerful phrase, and you don't even need the recording most of the time!

However, this then got me thinking of the alternatives suggested.

Email - keeping a record of the emails sent and received. I 100% agree this better, but not based on "not being able to use a call recording". I can take my time composing an email (I fail to do that some times, sadly). I can avoid losing my cool, etc. But then so can the other party (and they may have more training and practice).

My thought here is: how exactly is a record of an email exchange any different, legally, from a record of a spoken exchange. Why would one be "legal" and one not? Surely they hold the same "personal information" (if any). Does GDPR differentiate between them? Or is this not GDPR but something else? Surely the record of emails is just as much only allowed for "personal use"?

And how is that different than taking a screenshot of an "on-line chat" (another suggested idea). This even has the fact that the other party cannot simply assume that of course you have a copy (like email), as you had to actively make a copy, without asking.

And really, how is this different from old school written letters?

I am curious to know - where is the line drawn, why, and by what law?

Import Duty

Latest update on import Duty and VAT.

VAT is sorted for most couriers (well DHL and FedEx) as postponed VAT accounting. This is mostly automatic.

For duty we have an HMRC account now, all works. We advised DHL and they were helpful which was shocking. We have now received our first import delivery with duty, albeit £7.25. Emails from DHL about it. Statement from HMRC. I expect the direct debit soon from HMRC. ZERO ADMIN FEE from DHL.

This is finally clean, and no stupid admin fees. DHL for the win. Well done.

I have emailed FedEx and they are struggling to handle it, so yeh, not good. Hopefully we sort with them soon.

But DHL are still arseholes, sorry.

Before this import was one with £5.76 duty, which sort of kicked off sorting this all. They invoiced £5.76 plus a £5 admin fee.

To be clear, we have no contract with DHL, their contract is with the sender.

What is extra fun is when they told us, before delivery, we said "but we have a duty deferment account at HMRC" (not realising this needed setting up in advance with DHL), but also advising our published terms which state a charge of £20+VAT admin fee for payment demands from couriers.

Even after that they delivered - in my view that means they have accepted our terms.

Invoice for £10.76

However, being generous, and as they had actually been helpful getting the duty account all sorted, I confirmed we would pay the £10.76 and checked bank details for this. We paid it on 9th Nov, including the £5 admin fee we never agreed. We quoted the reference they asked us to.

Note, if you pay A&A with right reference it appears in your A&A account in seconds, you are immediately emailed a statement, and if clearing an overdue debt your services are instantly re-instated, 24/7 automatically. Handling incoming payments (especially if they have the correct payment reference) is not hard, honest.

Threat of legal action

Today (13th) I get a threat off legal action for £10.76. Yes, the £10.76 I generously paid (generous as it included £5 I never contracted for or agreed).

You can imagine the reply I have now sent...

Fucking arseholes!