[fixed] server unreachable

[Keith Smith]

Web site and forums are up (separate server), but flight server is unreachable. Looking into it right now.

[Keith Smith]

Here's the network status of our hosting provider for the flight data server: http://twitter.com/xlhost

They're working on it.

[Keith Smith]

power was apparently restored, but I still can't reach it properly. Am now working with tech support (was in line for quite a while, as you can imagine).

[Keith Smith]

coming up...

[Keith Smith]

and we're up. Post mortem will be posted shortly.

[Keith Smith]

Power was restored, however, the machine was not booting up correctly. It took a while to establish that we had power, but the machine wasn't reachable.

There was a separate hardware issue that had developed (possibly as a result of power outage, we're not sure) that required hands-on attention by one of the tech support specialists. Obviously, reaching a tech support specialist when there is a separate system-wide issue can be tricky, so we were delayed in getting the one-on-one help, but once we did, things moved quickly.

Had this gone on much longer, the plan was to flip the service over to a backup location. We were, in fact, in the process of getting ready to flip that switch when we learned that power was up and we had the separate problem. It was decided that we would stay the course and get the machine back up and running, a process which only took 10 minutes or so at at that point.

Apologies for the inconvenience. This was a very rare instance of an issue at this facility, something which hasn't happened for several years, according to two independent accounts from other sources.

[Mark Hargrove]

I hope you're going to get a more detailed explanation from them as to what happened. Your servers should be on UPS and the hosting facility should have their commercial power backed up by generators. The servers themselves should have redundant power supplies fed from separate PDUs on isolated circuits.

Power failures do occur, still, but they should be incredibly rare. "...hasn't happened for several years" meets my definition of "rare", but I'd still want to know exactly what happened. An extremely low-probability multi-point failure? (can't do much about those) --a cascading failure? (you CAN do something about those) --a procedural error by staff during maintenance? (again, CAN do something about those).

One reason it's good to know exactly what happened THIS time is so that you have a data point if it happens again within the next couple of years.

As a point of reference, a data center I was responsible for for many years has not had an unplanned power outage since it opened -- nearly 10 years now.

-M.