Royal Mail Shipping API - finally working

Since Royal Mail are pulling the plug on Smart Stamp, tomorrow, we have been rushing to get an alternative postage system in place. The obvious choice was the Royal Mail Shipping API, which they document on their web site.

This allows us to print postage, which is purchased "on account" rather than pre-paid on card like Smart Stamp.

The API uses XML and SOAP, and seems to be documented in the technical specification on the web site. There is an on-boarding platform to allow testing, and a check list of tests done before you can go live.

Frustratingly, in spite of a couple of months notice it took until 3pm the day before Christmas Eve before we got any credentials to test. All of my testing failed with "Authorisation Failure". Looking at the the documentation and the information they sent over this is clearly a common problem. There is a lot in the technical specification and they have code samples for generating the authentication strings in the XML in php, python, exe, and others.

However, today (yes, very last minute) we finally have this working on their live platform, so here is a list of the issues which I hope they can fix, and others can learn from.

Client certificate

Even though the https uses a self signed certificate their end, they want you to use a specific client certificate they issue. This means we'll have to faff with renewing it every year or so.

Why the certificate is not the only authentication is beyond me, or why, when using https, they do not simply use plain text http authentication or plain text SOAP authentication, is also a mystery.

Authentication

The authentication uses SOAP WSSE Password Digest authentication. This is well document, and involves sending a Created date/time, a Nonce random value to avoid replay attacks, and a Password which is a password digest. The digest is defined for WSSE as SHA1(Created+Nonce+Password) and both nonce and the digest are sent in Base64 in the XML.

What Royal Mail expect is: (a) Nonce is 16 byte binary value (b) "Password" as used in the WSSE hash is actually Base64(SHA1(PlainTextPassword)). This means you send Base64(SHA1(Created+None+Base64(SHA1(PlainTextPassword))))

Their documentation does not exactly make this clear at all, and contradicts itself,. Their example code does not do this, it uses just SHA-1(PlainTextPassword) as the password in the WSSE algorithm, so generates credentials that do not work. Their help desk would not actually make an XML file that works to the actual on-boarding system. They confirmed the examples I made using their own code fragments were "right" when there were not, and only used some SOAPUI test platform rather than using the actual on-boarding system. This wasted a LOT of my time.

I note that at least one example used a timestamp that was local time with a Z suffix (meaning UTC), so I half expect this to all go horribly wrong at the end of March.

ApplicationId

The application ID looks to be sort of your account number, a 10 digit code. But for some reason both the on-boarding and live systems cannot actually work with the right application ID, and helpfully just say authorisation failed. They have provided a different one to use for now. This just compounds the confusion over the authentication hash as trying either way of doing the hash still fails when using the right application ID.

Also, they quoted the application ID wrongly in the first place in their email, just to add to the fun.

Schema Errors

The XML you send is checked against the schema, and a single character out of place just says Schema Validation Failed. No clue where it got to. This is a nightmare to debug.

This is made worse by the specification actually quoting at least one object name incorrectly, so if you follow the spec you will get a schema validation error. For example, object unitofMeasure is actually unitOfMeasure.

XML namespace

Anyone that uses an XML library or understands XML namespaces will know how they work - each object and even attribute can have an associated namespace, which is itself usually a long URI.

These are then typically abbreviated to an arbitrary prefix. So soap:Envelope is saying that the Envelope object is in the soap namespace, but there is an xmlns:soap attribute somewhere at or above that object saying what the namespace actually is.

The tags you use are normally totally arbitrary, however, when I tried using different tags to those they used it failed validation! I don't know if this is only the on boarding system, but it means they are not doing XML right. Thankfully, using the tags they expect is not that hard, but it a requirement they should make clear.

It gets worse though, and this may only be on-boarding again, but the location of the namespace matters! Not for the schema validation, no, that would be too easy, but for authentication. My XML library places the namespace declaration as close to the outer most usage of that namespace. But doing that means you get the ever unhelpful Authorisation Failed error. This again compounds the confusion of the hash used. Thankfully my XML library has a mode that puts all namespace declarations on the top level, and that was good enough for the on boarding system (albeit not quite matching their examples).

Rather annoyingly they mix some objects using V1 of their API with some using V2. They also have subordinate objects that are in the default XML namespace (so untagged), which means you cannot simply make the whole message have an xmlns at the top level and avoid using tags. It is just messy.

Service Matrix

They have a service matrix - the services they offer are a combination of a service type, service offering, service format and enhancements, and then separate options for signature. It is actually really messy, e.g. for recorded delivery you use a service enhancement (a number) on a normal service list 1st class post, and do not set the "signature" field! Basically, we had to make a table of what things to set for each service we wanted to use. But we don't have a clear list of options from Royal Mail, i.e. other than just the title of each so not clear initially that CRL is Royal Mail 24 and Royal Mail 48 which are account (cheaper) versions of STL 1st and 2nd class! We hope to find out more next week, and something about the tracked but not signed for options which we want to use for routers.


I hope this is useful to anyone else doing this... P.S. I have C code for this - ask me on irc.