Royal Mail API (RMAPI v4)

We have used Royal Mail for postage for some time. They had an XML API which we have used, and it works.

What is interesting is how all of these APIs are moving from XML to JSON now. One of the big benefits of JSON is none of the FUCKING XML NAMESPACES (sorry, did I say that out loud?).

Even so JSON has its issues, but nothing like XML with namespaces.

Manuals

They have a web site with a login and loads of interactive stuff, and that includes the API and API manuals. They also have a "V4 Shipping API Consolidated Guide" PDF they sent. So how hard could it be. I had two manuals!

To put this in context, I am someone that has used (and written) many APIs that are JSON/http based over the years. I practically invented the concept (I even have a patent on form posting from over 30 years ago). I have some clue, honest.

I actually quite like the latest "web" based stuff with JSON sent and received. Even OAUTH crap. I have some experience with this, honest.

Update: Before logging in there are some extra links, and another set of manuals, which is yet again different and contradictory - arrrg!

Authentication

First challenge is authentication, and to some extent the manuals were not that bad. The one critical thing they could have done is include, in any of the manuals, one simple example.

They use OAUTH2 but in a way I have not seen anyone else use. I needed a POST to a URL with a URL encoding post of grant_type=client_credentials (and it had to be a POST not a GET) using Basic authentication of the client ID and key they allocated. This is an unusual way to OAUTH2, but valid, to my huge delay of some hours getting the hang of it was really down to me. I get back a Bearer key with expiry (1 hour).

API URL

One issue is the URL, well, the hostname. The documents state that the hostname to use is for their TEST system. This is good, they have a test system, but the other manual suggests it is the hostname for the LIVE system. Why?

I eventually worked it out, there is one hostname for authentication, and whether it is test or live depends on your account. Your account can be in "sandbox". I mean that is great, a good system, but why the fuck not state that in either of the manuals?

What adds to the fun is that hostname is actually only to authenticate. The actual hostname for the service is different, and searching both manuals I do not see the actual hostname for actual API calls listed anywhere.

Why? WTAF?

JSON blob

The big clue was some swagger.json file you can download. Loads of JSON, some "OpenAPI" thing. That has been way more helpful than either of the manuals!

It explains the actual hostname to use for the API, and each of the schemas for each API.

The manual, well one of them, lists fields you need, but omits that these may or may not be in sub objects. The JSON blob explains in way more detail.

And, of course, this JSON file, whilst really good, is missing a load of things, like creating manifests!

Errors

For example, the error "Could not find member 'ContactName' on object of type 'Destination'. Path 'Destination.ContactName'" fooled me for a while as I was supplying Destination.ContactName. What it meant is I should not be supplying that. I read "Count not find" as I was missing something, but no, it meant I had supplied something it "could not find" a use for. That is shit UI. (FYI I should have provided Destination.Address.ContactName)

Tracking

There is, of course, tracking. There is an API to get tracking and an option to have tracking posted to a web hook - yay!

But the documentation for the tracking API just says the response is a tracking number? Which, err, makes no sense, what of the status of the shipment. Hmmm.

And apparently the tracking web hook is not supported, so why the hell is it in the API documentation? Arrrg!

Working

I have it working now - but this is literally one of the worst documented APIs ever.

The raw JSON file was more documentation than either if the manuals, but even that is missing bits.

Icing on the cake

Royal Mail have What3Words embedded. I kid you not.

This is so mental! RM have PAF which has detailed (like, to 10cm) delivery point details on all UK addresses. They are the one organisation that has ZERO use for W3W.

I don't know what they do with it - I will ask?

• If a W3W supplied is way off, like hundreds of miles, do you use it?
• If a W3W is next door to postal address provided, do you use it?
• If a W3W is within target address, e.g. a shed in garden, do you use it?

If all are "no", I have to ask WHY THE FUCK it exists in their API!

[Update: Answer is they don't use the W3W filed.... So WTF have it?!?!]

WIFi: Aruba vs FS

I have used Aruba WiFi for some time - they are nice, but expensive. The latest Aruba Instant series seem more sensibly priced though.

When picking WiFi there are a number of factors, but if you are going to get more than just the WiFi that comes with your broadband router you almost certainly want something that can allow multiple access points and roaming between them.

There is also a matter of the number of antenna and radios, and the protocol - summarised as catchy names like WiFi5 and WiFi6, these days. There are also access points for outdoor use, which can be useful.

The Aruba has one cool feature, and the main reason I like it, that it does not need a controller. A controller is what directs devices to move from one access point to another as you move around, and what manages the overall configuration. With the Aruba, the controller is one of the access points, and can even fall back to anther one if the main one is off.

So what changed

I have been pretty happy with the Aruba for many years. It has a lot of features and settings (perhaps too many), and as I say, needs no separate controller.

But recently some of my IoT kit (ESP32), which is all 2.4GHz, has been playing up, sometimes sulking not reconnecting to WiFi, and so I felt it was worth changing the WiFI to see if it helped.

Long story short, the WiFi choice was not the cause of that specific problem, and I think I have fixed it. But the change of WiFi had some surprising side effects. For reasons I won't go in to, I have all my 2.4GHz on the same channel. And so that is a pretty full channel. As such I was never surprised to find s/w upgrades on a set of IoT devices was always a bit slow - I set up to do one at a time. To my surprise, with the new WiFi, it is way quicker!

What is the new WiFi?

I have got some kit from FS. It is cheaper than the Aruba, which is a start, but it needs a controller. The controller (AC-1004) is also a 4 port PoE switch which makes it actually useful for a small home installation as you almost certainly want a PoE switch for your access points anyway. So again, not that much extra cost. It can handle 64 APs. I may be a convert to the idea of a controller.

The (web based) controls are a but more clunky than the Aruba, but to be honest, pretty similar. It also has a huge command line interface!

The APs actually come with mounting brackets, unlike the Aruba! I have AP-N505 indoors and AP-T565 outdoors.

So far, so good. I'll see if I am cursing the change in a few weeks. If not, I'll take down the Aruba APs.

BMA423

The BMA423 is a small accelerometer by Bosch. The data sheet is here. It is 129 pages. It is really crap if you want to just use this chip. It repeats itself quite a bit as well. The Quick Start Guide has a nice vague bit about writing the init file as a burst write to address 0x5E.

There is reference code, and the help forums for this just have Bosch saying to use the reference code. Some of us would rather not. A good data sheet should not expect you to just use the reference code. It is also a bugger to follow.

One of the main issues is the the reference code works with registers 0x5B and 0x5D which the data sheet lists as reserved. This is silly, the data sheet could define these. The reference code also has 6kiB of binary code that seems to need to be sent.

So, here is what I worked out, and it is now working.

Initialisation

Reading and writing registers is standard I2C register read write. I.e. write the register address and then data, or write the register address and then read data, allowing multiple byte writes or reads as normal (with on exception, see below).

Checking

• You can check all is well, and init done, by reading 0x2A (INTERNAL_STATUS), and confirming it is 0x01

Soft reboot

• You can do a full soft boot by writing 0xB6 to 0x7E (CMD).
• Wait 1s after soft boot

Loading ASIC code

• Write 0x00 to 0x7C (PWR_CONF). This disables aps (advanced power save).
• Wait 500ms
• Write 0x00 to 0x59 (INIT_CTRL)
• Send the 6k block to the ASIC address 0 (see below) - see here for the data.
• Write 0x01 to 0x59 (INIT_CTRL)
• Wait 150ms
• Check status by reading 0x2A (INTERNAL_STATUS), it should now be 0x01
• Read the ASIC address at 0x5B/0x5C, this is where you load FEATURES_IN (see below)
• Write 0x01 to 0x7C (PWR_CONF). This re-enables aps

Block read/write to ASIC

It seems there is a separate ASIC block of memory, and this is accessed by three registers.

• 0x5B is low byte of address to next read/write - BUT NOT AS SIMPLS AS THAT!
• 0x5C is high byte of address to next read/write
• 0x5E is the gateway to reading and writing

Normally, with I2C it would auto increment the register on read/write when transferring multiple bytes. This does not happen with 0x5E. You can write to 0x5B/0x5C as a 2 byte transfer, as per normal I2C.

The ASIC address is stored oddly. It has to be even (bit 0=0) and you have to read/write even number of bytes at a time via 0x5E. The address is stored such that 0x5B is (address>>1)&0xF, i.e. bits 1-4 of address. 0x5C is address>>5 (i.e. bits 5-12). So you need to allow for this when reading or writing the ASIC address.

It seems from reference code you can transfer up to a maximum of 8 bytes at a time, so you have to repeatedly write the ASIC address to 0x5B/0x5C then write 8 bytes to 0x5E in one block. However, my testing suggests you can transfer the whole 6kiB init in one go, so totally pointless reference code.

Once the init file is written, and the status is 0x01, you have to read the address in 0x5B/0x5C which tells you where the FEATURES_IN block is located in ASIC memory... Shift it appropriately to get the byte ASIC address.

FEATURES_IN

FEATURES_IN is block of 64 bytes in ASIC memory. You can read/write even number of bytes, setting 0x5B/0x5C to the base address (as read above) plus the offset in to the block. The data sheet does list these additional registers reasonably well. It just lacks any explanation of how you access these bits!

You can read all 64 bytes in one go, and write all 64 in one go, or read/write even numbers of bytes if you only need to change one register.

Steps

To enable step counting, I have done the following :-

• Write 0x00 to 0x41 (ACC_RANGE) to set ±2G
• Write 0x02 to 0x56 (INT1_MAP) - only needed if you want interrupts
• Write 0x04 to 0x7D (PWR_CTRL) to enable accelerometer
• Set FEATURES_IN(0x37) OR'd 0x30 to enable activity and counter
• Set FEATURES_IN(0x3B) OR'd 0x10 - not idea what this does as reserved in data sheet! 

Reading steps is then 4 bytes from 0x1E onwards, low to high bytes for 32 bits.

Resetting the step counter is setting FEATURES_IN(0x47) OR'd 0x04 (self clearing).

NON-STERLING TRANS FEE

One thing pisses me off is cards charging a Non sterling transfer fee. I'd have expected them to make money on the exchange rate, or be big enough that they can handle currencies without such silly fees to be honest. Even more so when it is someone like American Express, that, well are American, and I am paying something in American dollars. You would think they could handle dollars.

But I have recently discovered something slightly odd.

VISA

Using my VISA debit I was charged...

AMOUNT IN USD 206.87 ON 28 JUN VISA 1.269999 FINAL GBP AMOUNT INCLUDES NON-STERLING TRANS FEE £4.48 = £167.37

But when I was then refunded, it looks like...

AMOUNT IN USD 206.87 ON 30 JUN VISA 1.2669 FINAL GBP AMOUNT INCLUDES NON-STERLING TRANS FEE £4.49 = £158.80

This means I was charged for the refund, and oddly charged at a higher rate (by 1p).

Amex

Using my Amex I was charged... (different day, I have not compared exchange rates)

71.15 UNITED STATES DOLLAR, Non-sterling transaction fee: £1.68, Exchange rate: 1.2687 = £57.76

But when I was then refunded... (next day)

71.15 UNITED STATES DOLLAR = £57.82

Comparison

VISA charge the non sterling fee on charge and refund, but Amex do not. Indeed it seems they effectively refund the non sterling admin fee.

VISAs fee appears to be around 2.7%, Amex fee appears to be around 2.9%, very similar.

Conclusion

It is actually in my companies best interests if I pay for these on my personal Amex and claim on expenses, if there is any chance of a refund (which happens quite a lot if I have arguments with JLCPCB, for example).

toot

Twice I setup Mastodon.

Once many years ago - I followed the step but step guide and got it working. It worked for quite a while and then broke - out of disk. I gave up.

Then a second time we easier - this time my ops team set me up a VM, and Mastodon had improved a bit, so set up again, following the instructions. Again it worked. It caused problems as it was a "new" instance on an existing domain, and that caused things that had cached stuff to get confused. It got sorted, people had follow requests they had to cancel and try again, that sort of thing.

These days it seems to have a lot more "cleanup" logic, and a lot more obvious on disk usage, but still far from perfect from what I can see.

WTF is it not simple?

My big issue really is that Mastodon should be like apache, or many other quite complex systems, which are basically apt install mastodon. That really is what it should be. Simple as that. The web interface or command line could take you through the initial steps for admin user password and domain name and so on, but that should be it.

There are docker instances. I know nothing about docker, but I am told they sort of just work. Maybe that is the answer?

But why is it not as simple any anything else on linux, really. It seems Ruby is the main problem, as is python it seems. A pain in the arse.

But all is well, so not problem.

Yes, all is well, and even upgrading to next mastodon version as needed was not that hard. Again, following a load of step by step instructions not a simple apt upgrade mastodon or some such. But not rocket science.

But err, then, it broke. Basically the ops team wanted to upgrade Debian to bookworm, which is fair enough, but doing so cleaned out almost all of the stuff Mastodon needs.

Now I am not just upgrading Mastodon, and not installing Mastodon (as I don't want to lose/break what I have), I am sort of trying to make it work from instructions that are one or the other, and constantly hitting stupidities like OpenSSL is not available (how the fuck is that not standard in these tools?!) and not clue how to fix that.

I gave up

I must be getting old, because, yes, I gave up, it is too much of a pain in the arse, and I have better things to do. I tried, honest, but life is too short.

Bring in the experts!

The simple answer is to call in some experts. In this case Mythic Beasts. They actually run the A&A instance because my ops team were smart enough to not go near this with a barge pole.

They are not only running my instance, but in very short order managed to copy all the stuff from the old, and now broken, instance over, and make it continue seamlessly for me, well mostly - some avatar issues they are fixing. But followers, and so on, all working. I have not missed anything and was off for a day at most.

It is not cheap, it is several hundred quid a year. But what do I value my time at?

There is an irony here. I am paying that for my social media. If the likes of Twitter had people paying that for a really good, clean, not sales/advertising/pushy/confrontational/stressy service, they would make a fortune. That is not their business model.

The only mitigation is it may end up being more than just me - some family members, when they finally catch this boat. Of course, subject to whatever stupid rules the Online Safety Bill may create, LOL!