Yesterday was rather busy - starting with a power outage at the data centre the night before.
Whilst the outage was under 4 minutes, and equipment should all start up and carry on after an outage, it seems four servers did not. This makes me wonder if it was a power spike, but I have not seen the report from Pulsant yet.
Most things did start working, as you would expect, and our routers recover within a couple of seconds of power being restored. One server that failed was a spam filter, so just meant we had less capacity for spam screening. Another was one of the outgoing mail server pool, and again we can work with one being off line. Finally one of the disk servers (with all my stuff on it, as well as customer voicemail recordings) was down. One server handled usage recording for calls and broadband, so people got some for free!
Jimi and Paul worked on these issues over night, and Paul and Andrew were working on them during the day. The mail server queue was cleared. The disk server fixed. I was able to ensure the billing system was working correctly once the usage recording database was back. It was a busy day all around.
The day did not end there as something blipped last night in the other data centre with two routers having trouble with a switch. Thanks to Paul and Jimi for work on that and fixing it within a few minutes. Again, most systems re-routed automatically, as they should.
Well done everyone.
Subscribe to:
Post Comments (Atom)
Playing with microphones
The latest LED board designs have included a TDK PDM I2S microphone - the idea was to make sound reactive LED strips. It is tiny (3.5mm x 2...
-
Broadband services are a wonderful innovation of our time, using multiple frequency bands (hence the name) to carry signals over wires (us...
-
For many years I used a small stand-alone air-conditioning unit in my study (the box room in the house) and I even had a hole in the wall fo...
-
It seems there is something of a standard test string for anti virus ( wikipedia has more on this). The idea is that systems that look fo...
No comments:
Post a Comment
Comments are moderated purely to filter out obvious spam, but it means they may not show immediately.