TOTSCO Tick Tock

I may sound like a stuck record, but I learn more as I go, so updating on this seems sensible. I hope it helps other CPs.

TOTSCO do publish the test process, here. But I'll summarise.

• A simple connectivity test, to check connectivity, and send some dummy message responses.
• Integration testing with another CP, exchange a number of messages of different types.
• Production Implementation testing.

To summarise the problems so far.

Simulator

It is meant to do basic connectivity tests, but did not actually pick up the one issue we had that we were too slow responding (a simple apache config tweak fixed). So failed in its one job.

We could not complete tests as the responses were not valid. Heck, we had to bodge things to even send a message as the simulator does not meet the spec for the URLs used. This meant we would not then send messages to progress a switch order, as they wanted, because we had (correctly) rejected the invalid messages they had sent, and had no switch order to progress. Thankfully that step was not mandatory.

Integration Testing

This purports to be more comprehensive testing, but it has a lot of issues.

1. It is testing with a buddy CP, but it seems it can take weeks to have one assigned, at random. We short circuited this eventually be agreeing with another CP we know. But they were not ready. What is worse is that we now know that at the same time as we are waiting weeks, other CPs are as well, which makes no sense?!?!
2. The buddy CP is doing testing with us, for free(!), if they are ready. They may not be ready. Even if they are, we are doing tests against their interpretation and implementation of the OTS specifications. It is not testing to a reference implementation or against the specification.
3. They want us to do all 15 message types (plus the 16th messageDeliveryFailure). One issue is that this is contrived. Our system is design to send valid messages as part of a switching process, integrated in to our management systems (with just the tiniest tweak for testing to not actually action a cease or migrate at that key point). This means getting failure response to some messages will not use our normal system, because our normal system would not send incorrect messages.
4. They then want at least 1,000 messages - why? These are not real switch orders. It will literally be a repeated match request sent 1,000 times. A totally pointless step.

I have had to make the system allow me to send bad messages in some ways in order to get the Failure responses. This means I am not testing the actual OTS system we have made, I am bodging it! If there was a proper test system, one could set up the bad responses even for valid messages so as to test, and to generate bad incoming messages to test error checking. But if you have two CPs that have set up systems correctly, they would not generate bad messages and therefore prompt error responses as a result. You actually need a buddy CP that is set up to deliberately do testing, for free(!).

This is where we are still - I asked for another buddy CP a few weeks ago, and no joy yet, but the original CP may be closer to being able to do basic tests now. I hope so. It could allow us to finally get past this step.

Production Implementation testing

This is the final step before able to go live.

1. We have to book a test slot 8 weeks in advance. Why in the name of sanity would we have to do that? I mean if the test slot meant tying up TOTSCO staff for hours to go through a series of complicated tests, I could understand - but this is the kicker...
2. The test is one message exchange. Just one. I don't see how this even takes up TOTSCO staff time. It should not. It could be automated - I fill in details - I send a match request to BT or someone, and get a response, done. Why on earth is a test slot even needed in the first place, let alone booking 8 weeks in advance.