Since Royal Mail are pulling the plug on Smart Stamp, tomorrow, we have been rushing to get an alternative postage system in place. The obvious choice was the Royal Mail Shipping API, which they document on their web site.
This allows us to print postage, which is purchased "on account" rather than pre-paid on card like Smart Stamp.
The API uses XML and SOAP, and seems to be documented in the technical specification on the web site. There is an on-boarding platform to allow testing, and a check list of tests done before you can go live.
Frustratingly, in spite of a couple of months notice it took until 3pm the day before Christmas Eve before we got any credentials to test. All of my testing failed with "Authorisation Failure". Looking at the the documentation and the information they sent over this is clearly a common problem. There is a lot in the technical specification and they have code samples for generating the authentication strings in the XML in php, python, exe, and others.
However, today (yes, very last minute) we finally have this working on their live platform, so here is a list of the issues which I hope they can fix, and others can learn from.
Client certificate
Even though the https uses a self signed certificate their end, they want you to use a specific client certificate they issue. This means we'll have to faff with renewing it every year or so.
Why the certificate is not the only authentication is beyond me, or why, when using https, they do not simply use plain text http authentication or plain text SOAP authentication, is also a mystery.
Authentication
The authentication uses SOAP WSSE Password Digest authentication. This is well document, and involves sending a Created date/time, a Nonce random value to avoid replay attacks, and a Password which is a password digest. The digest is defined for WSSE as SHA1(Created+Nonce+Password) and both nonce and the digest are sent in Base64 in the XML.
What Royal Mail expect is: (a) Nonce is 16 byte binary value (b) "Password" as used in the WSSE hash is actually Base64(SHA1(PlainTextPassword)). This means you send Base64(SHA1(Created+None+Base64(SHA1(PlainTextPassword))))
Their documentation does not exactly make this clear at all, and contradicts itself,. Their example code does not do this, it uses just SHA-1(PlainTextPassword) as the password in the WSSE algorithm, so generates credentials that do not work. Their help desk would not actually make an XML file that works to the actual on-boarding system. They confirmed the examples I made using their own code fragments were "right" when there were not, and only used some SOAPUI test platform rather than using the actual on-boarding system. This wasted a LOT of my time.
I note that at least one example used a timestamp that was local time with a Z suffix (meaning UTC), so I half expect this to all go horribly wrong at the end of March.
ApplicationId
The application ID looks to be sort of your account number, a 10 digit code. But for some reason both the on-boarding and live systems cannot actually work with the right application ID, and helpfully just say authorisation failed. They have provided a different one to use for now. This just compounds the confusion over the authentication hash as trying either way of doing the hash still fails when using the right application ID.
Also, they quoted the application ID wrongly in the first place in their email, just to add to the fun.
Schema Errors
The XML you send is checked against the schema, and a single character out of place just says Schema Validation Failed. No clue where it got to. This is a nightmare to debug.
This is made worse by the specification actually quoting at least one object name incorrectly, so if you follow the spec you will get a schema validation error. For example, object unitofMeasure is actually unitOfMeasure.
XML namespace
Anyone that uses an XML library or understands XML namespaces will know how they work - each object and even attribute can have an associated namespace, which is itself usually a long URI.
These are then typically abbreviated to an arbitrary prefix. So soap:Envelope is saying that the Envelope object is in the soap namespace, but there is an xmlns:soap attribute somewhere at or above that object saying what the namespace actually is.
The tags you use are normally totally arbitrary, however, when I tried using different tags to those they used it failed validation! I don't know if this is only the on boarding system, but it means they are not doing XML right. Thankfully, using the tags they expect is not that hard, but it a requirement they should make clear.
It gets worse though, and this may only be on-boarding again, but the location of the namespace matters! Not for the schema validation, no, that would be too easy, but for authentication. My XML library places the namespace declaration as close to the outer most usage of that namespace. But doing that means you get the ever unhelpful Authorisation Failed error. This again compounds the confusion of the hash used. Thankfully my XML library has a mode that puts all namespace declarations on the top level, and that was good enough for the on boarding system (albeit not quite matching their examples).
Rather annoyingly they mix some objects using V1 of their API with some using V2. They also have subordinate objects that are in the default XML namespace (so untagged), which means you cannot simply make the whole message have an xmlns at the top level and avoid using tags. It is just messy.
Service Matrix
They have a service matrix - the services they offer are a combination of a service type, service offering, service format and enhancements, and then separate options for signature. It is actually really messy, e.g. for recorded delivery you use a service enhancement (a number) on a normal service list 1st class post, and do not set the "signature" field! Basically, we had to make a table of what things to set for each service we wanted to use. But we don't have a clear list of options from Royal Mail, i.e. other than just the title of each so not clear initially that CRL is Royal Mail 24 and Royal Mail 48 which are account (cheaper) versions of STL 1st and 2nd class! We hope to find out more next week, and something about the tracked but not signed for options which we want to use for routers.
I hope this is useful to anyone else doing this... P.S. I have C code for this - ask me on irc.
Subscribe to:
Post Comments (Atom)
Playing with microphones
The latest LED board designs have included a TDK PDM I2S microphone - the idea was to make sound reactive LED strips. It is tiny (3.5mm x 2...
-
Broadband services are a wonderful innovation of our time, using multiple frequency bands (hence the name) to carry signals over wires (us...
-
For many years I used a small stand-alone air-conditioning unit in my study (the box room in the house) and I even had a hole in the wall fo...
-
It seems there is something of a standard test string for anti virus ( wikipedia has more on this). The idea is that systems that look fo...
If you have been reading some of the other threads around this you may be aware that I have struggled through this as well. To follow on for RevK's post I also have it working in VB.net with the application ID assigned to our account. I will put this up on github in the next couple of days once I have finished the library, ill pop back and post the link.
ReplyDeleteHi Dave, I'm curious, have you uploaded this to GitHub yet? I'm reviewing the API documentation, planning to write integration code, however, if someone else has already done the heavy lifting, I'd be happy to use it.
DeleteI would also be very interested! - I've been banging my head against a wall for so long trying to figure this out. I'm getting schema issues whereby my request doesn't quite match theirs. I think it's to do with the prefixes they are using, and my WSDL/soap output isn't using them the same.
DeleteHi Dave, as another frustrated customer of RM (the API documentation is a joke), if you could post the VB code it would me much appreciated. I'm starting to dream of E0007 Authorisation failure!
DeleteThis comment has been removed by the author.
DeleteHi Dave, i have wasted many hours trying to make a call to royal mail using the shipping API without success, i would greatly appreciate any VB code you can share to help me proceed.
DeleteCheers
Hi Dave,
DeleteLooks like you have not got the code to github yet. Did you have problems with it? Does anyone know anywhere where there is some vb.net or c# example code that I could start with?
Cheers
Dave, Thank you for your insights which have helped. Doing this API is VERY PAINFUL, RM do not make it easy, as of June 2016 there is STILL things not in the Spec.
DeleteDave, Thank you for your insights which have helped. Doing this API is VERY PAINFUL, RM do not make it easy, as of June 2016 there is STILL things not in the Spec.
DeleteHi all - I got code working to the point of being able to communicate with them - getting schema validation error, but I managed to figure out what I was doing wrong with digital certificate
ReplyDeleteSo 1 step forward :-)
Programs depending on the namespace tags, rather than the URI, is so common that you might as well start out by assuming that is what's happening in any new system you see. This tends to mean that their underlying processing doesn't support namespaces at all -- they've just got hardwired tag names that happen to have colons in them (talk about missing the point...)
ReplyDeleteSo now I am onto altering namespace - grrrr
ReplyDeleteI am overriding EndpointBehavior etc etc
but the class that I was using to disply the xml sent stops working correctly now
would that be expected?
will I have to use something like fiddler?
Sounds like VB or NET and I have no idea on that, sorry. Our stuff is all C code on linux, which I am happy to share.
DeleteHi - can you supply a link please. I am really finding this difficult.
DeleteThank you for that offer - I eventually worked out why
ReplyDeletethe BeforeSendRequest is called before the 3 new class overrides start to deal with the message and do replaces
So I downloaded Fiddler - struggled with it
So I downloaded Charles - and hey presto
I can see the XML outgoing soap (which really does help when trying to get such exact xml)
Now that I am getting there, I feel I can keep going, and I am now down right determined to get it to work
So thank you for offer of code - if I get utterly stuck I may take you up on that, but right now I am enjoying the glow of small triumphs
Royal Mail phoned today and my partner passed on my unhappiness
Sadly I was at work, I would truly have loved to pass on some messages in person
They actually want to send someone to show us how to use DMO
thanks but no thanks.....think I can figure out the website just fine
Big Grin
ReplyDeleteGot it all working :-) kinda chuffed since it's been 15 years since I did any vb.net
have even managed to automate sending the PDF direct to the zebra printer
So, just bought Dymo USB postal scales - next little challenge - to pull weights directly into application
:-)
Hi, as someone getting nowhere fast with the API would you be able post the VB.net code? It would be much appreciated, I'm starting to dream of E0007 Authorisation Failure!!
DeleteHi
DeleteI have got to the stage of the dreaded namespaces using VB.net, I have created the 3 classes but now seem to be stuck again, where is the BeforeSendRequest located that you mentioned above? maybe this will get me moving on again :)
This comment has been removed by the author.
DeleteHave 1 question
ReplyDeleteDigital certificate prompts for password - is it possible or advisable to provide this in code to remove need to enter password?
That is one to consider as security policy - if the password is in the code, or command, or whatever is that any more secure than having no password on the certificate? If the whole thing is pretty safe, locked down to what user runs it, etc, then maybe no need for password. But I cannot really say for your situation.
DeleteIs there any experience here with the new V2 of the API which supposedly does away with the need for a certificate and instead uses a ClientId and ClientSecret which are, according to the documentation, to be sent as HTTP Headers of X-IBM-Client-Id and X-IBM-Client-Secret.
ReplyDeleteI am using a basic WCF client created from the wsdl but whatever settings I try I cannot get through the error "The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was 'default'."
Has anyone had similar experience with the new API ?
Not heard of a new API. Sounds almost as bad!
DeleteDid you ever resolve this issue Roger?
Deletei'm having the same issue at the moment and would be grateful for any help you can provide if you have already solved it
Full C# Royal Mail Shipping v2 solution available here: http://stackoverflow.com/questions/38896117/royal-mail-shipping-api-v2-soap-authentication-c-sharp
DeleteI just solved the unauthorized issue. You would need include X-IBM-Client-Id and X-IBM-Client-Secret in the request header. I used the method in this link solved the problem: http://stackoverflow.com/questions/38896117/royal-mail-shipping-api-v2-soap-authentication-c-sharp
DeleteHi Rev,
ReplyDeleteThis has helped me out a lot! did you ever come across what the wsu:id was for in the usernameToken tag with the ws-security?
I think we made shit up for that!
DeleteGood articles. Struggled with this and C# for days back in July. I posted some of my solution on SO a few months back, which might be of help to people reading this blog.
ReplyDeleteThe namespace issue was the biggest problem for me which was solved using the following: http://stackoverflow.com/questions/31595770/c-sharp-wcf-namespaces-move-to-header-use-ns-prefix/31597758#31597758
I then posted some C# code in response to someone else struggling with this which should be enough to point anyone in the right direction: http://stackoverflow.com/questions/34508811/consume-wcf-royal-mail-api-in-c-sharp-console-application
It looks like this API is for parcels only - did you ever come across an API for standard letters (1st / 2nd class)?
ReplyDeleteSo for all I've found is their online web print service (https://www.royalmail.com/discounts-payment/online-postage/home) which has no API. Really want to print labels and postage directly if possible.
No, does letters at franking rates as well!
DeleteAwesome! They should make that more obvious on their website! Franking rates without the need for a franking machine, contract, headache!
DeleteLet the integration commence!
Wow, would even do the bloomin' Christmas cards with this. Excellent!
DeleteHi RevK - do you have a git available of your code? or details to contact you on IRC? We are setting up our store back end and require the API for our team to use.
ReplyDeleteI can email it if you need - though they are talking of changing the API for some reason.
DeleteThat would be great thanks, if you could send it to ewan at relchron dot com that would be much appreciated. The Royal Mail are a law unto themselves ;)
DeleteHi RevK,
DeleteI'd be grateful for the code as well at scott at canuckconsulting dot com scott@canuckconsulting.com. Having an absolute nightmare getting this working!
Thanks for taking the time to post about it.
Scott
Hi Revk
DeleteSorry to trouble you. Could you please kindly email me the git for the code to anis at afuaa dot co dot uk.
Much appreciated and thank you for sharing.
Anis
HI RevK
ReplyDeletehope you can help, do you have a git available for your code? If you could please kindly email it if one available to anis(at)afuaa(dot)co(dot)uk... Thank you
I struggled with this recently, and I wanted to do it within the SOAP structure .net uses. So I wrote down all my steps :) Including a fix for a new issue that just appeared today.
ReplyDeletehttps://medium.com/@tonycheetham/royal-mail-soap-api-v2-interface-in-c-98081cf8ac4e