Peer1 had a major problem with the Vancouver area power failure this
morning (Monday July 14th). Their backup generators wouldn't carry the load. That took
from about 11am to 4pm to fix. Now it is our turn to try to pickup the
pieces and get everything back up and running, but that is proving a bit
more difficult to do than it should be.
As of 7:30pm: star3 is still offline, we're waiting for assistance from peer1
to get that resolved, and of course, they've got lots of clients needing
assistance. The server that handles mail for the NT machine is also waiting
help from peer1. The internal baremetal server that handles domain
registrations, payments, etc is also offline... that one we are
attempting to transfer to a stable box, but that may take a while.)
8:30pm: the hardware underneath star3 and the mail server for the NT box is
not going to come up without someone pulling the lids off, which depending on
how I fare restoring from backup may happen tomorrow morning. Note, that
support@baremetal.com won't be operational until the same box handling
registrations and payments is back up.
9:30pm: ok, star3 is going to need to be restored/rebuilt from backup,
that means moving 100+ gig of data around, so it will be a few hours yet.
The internal baremetal server has been moved to a stable box, so
registrations, payments, and email for support@baremetal.com are back.
E-mail for the NT accounts will probably be the last of our priorities
(sorry).
midnight: Still working on star3. I've got a restore from backup occuring
onto a partition on a big xen server, and I'm still trying to get the
original machine to boot. We're working with Peer1 on a "rescue floppy".
Getting the original machine back would be a lot simpler, as there would
be no data loss.
1:30am: Star3 is back up. It is the original machine, so there should be
no lost data, lost mail, etc. Obviously, the box was offline for a while,
so there will be delayed mail, bot nothing should have been lost. So that
just leaves the mail services for the NT box (14 accounts), oh, and the
audio server. We'll get to those tomororow. Both are going to require some
special techniques to get an older linux kernel to run. -Tom
4pm tuesday: audio is back. I did it before the NT mail services box
because I thought I would be
using the same technologies to get the old linux 2.2 kernel which had
very good transparent proxy capabilities running. That may or may not
end up being true... Sorry, for the delay. Have been working hard at it.
-Tom
The NT mail services box came back into service around 11pm on Tuesday.
We may make more changes for performance reasons, but everything should
be running ok now.
