We have a change freeze, started this afternoon, and going on until after new year bank holiday.
The principle is fine - we have a lot of staff off, especially some of the senior technical staff, and none of us want major issues whilst we are at home with family if it can be avoided.
So the idea is we don't make major changes or deploy new systems over the change freeze. Nice idea.
There are, however, a few problems, and it is a change for the way I work for a start.
I am very keen to do a job and finish and deploy it - I hate having any job interrupted by a big gap - I lose track of what I am doing and spend a lot of time catching up and things can be missed. So this means that where there is a job in progress before xmas, I have been rushing to make sure it is all deployed before the change freeze. This is not to say taking short cuts as such, but rushing. I don't want a half finished job not deployed. And no, finishing on a dev system and deploying next year is not good - I like to deploy things as I go and pick up issues and fix them whilst still fresh in my mind. If I did the work and did not deploy for two weeks that would be horrid.
We also have the fact that xmas can be a quiet period from a technical point of view - it is (was) an ideal time to deploy and test changes with lower than usual impact. For a start, a whole bunch of customers are not even there - businesses shutting down. And whilst I don't mean to say business customers are more important than residential, there is a difference for a business customer disrupted in their business for a few minutes, or a home customer disrupted whilst eating mince pies. So traditionally the xmas break has been a good time to work on some major projects and iron out the bugs before everyone is back to work or doing anything serious with what we sell.
Ironically, whilst a few months ago, I would almost be happy to sit around doing nothing all xmas break, we now finally have me on medication for my blood pressure which has had some sort of impact on my diabetes which means for last few weeks I feel much more like I was in my 20's, bored of watching TV, and coding from dawn to dusk (well much later with it being winter). Seriously, this is great, even if it won't last (5mg perindopril + 1.25mg indapamide, FTW).
We had only one snag with stuff rushed through yesterday, and it was not actually due to rushing at all, it was a VoIP issue, which is a complicated set of issues where a recent change, which had been tested on several boxes, was deployed as part of an urgent update to address a customer issue. Sadly, when load got to a certain magic level on the live VoIP servers we go drop outs. Our normal testing on several other boxes did not pick it up, and would not have had we not had the change freeze and hence done the update next week instead. Sorry for the inconvenience on that - the VoIP servers are a pain as reloading means dropping calls but waiting means people with dropouts in calls until we do - we managed to move calls and so only drop a few to get the new code deployed during the afternoon.
But overall I feel rushed by the change freeze and not entirely convinced it will help with issues cropping up or not. I guess we'll see over next couple of weeks, if I go crazy, and/or make a huge list of changes all done on Jan 3rd and consequences of that.
If I do have my mojo back, I am damn well going to do stuff, but maybe not A&A stuff. My son has a load of web/app sites that could be tidied up, and my mate Mike has loads of stuff he wants re-inventing from scratch (probably including "the wheel", knowing him). I may find stuff to do.
So, happy freeze everyone.
Speaking as someone who works remotely over the Christmas period (and thus cannot e.g. even restart a router if things go wrong), I have always been astonished that A&A rolls major changes out when nobody is around to fix any problems that might happen. It's even more astonishing to me that you find code freezes over "dead periods" like this to be surprising: it's been de rigeur to avoid doing anything disruptive at such times in the networking and software development industries for literally decades by this point, quite possibly for longer than I've been alive.
ReplyDeleteWell historically I have been arounds to sort things if nothing else, and so have many others.
DeleteI wonder if Nick's point was more that, if a change by A&A required customers to reset their routers, fewer people would be in the offices to do their resets?
DeleteThat, or anything else that needed to be done on the customer end. Or if something went wrong and nobody was around that was fixable and was fixed, but people were disrupted nonetheless because they were trying to get into their system remotely and were stopped from doing so, because, say, they were on holiday but had exploited remote working to work from their holiday site (via their home system) for a couple of days in the middle of the larger holiday. (I know two ex-A&A customers who left for precisely this reason, so this is not theoretical... of course, one of them is back because everyone else is worse!)
DeleteAt work, we have a code freeze for the opposite reason. We had the busiest weekend of the year 2 weeks ago on Black Friday, and the Christmas period is still our busiest month.
ReplyDeleteWe do it slightly different; some of our more technical customers are given the option of paying a little less for their service in exchange for their connectivity being terminated on our backup LNS.
ReplyDeleteWe test all new code releases and significant config changes on there with the understanding from our customers that, if asked, they will assist us with debugging any issues.
Once our 'canaries' are happy, we roll those same changes out to the primary LNS to all the other customers and to ensure that our primary/secondary LNS behave identically in case of failover as both LNS' are located in different PoPs.
We have a separate test LNS which usually has a few customers on in for just that, but it is not really what the change freeze is about.
DeleteI have to say when I was still working professionally, I liked Christmas for code change, as it was quiet I am Scottish and I could work without the hinderance of colleagues advising me. However, as a fellow "person with type 1 diabetes", I find the Indapamide effect has lasted. Perindopril, Amlodopine and Indapamide combined to revive me about 5 years ago and the effect has lasted! There is hope yet, now if only I could reduce my insulin.
ReplyDeleteThat is good to hear. When they started me on the indapamide I had to nearly double my insulin - it was weird.
DeleteLate as usual... If you need to bounce the VOIP server software, why not have a second, corrected instance signal the current running software to drop the server listener, so it can be started on the second instance. If you are only allowed a single instance to communicate to the carrier, then have it go through a proxy on the same box.
ReplyDelete