Server Crash, Updated

I logged in this evening to check my mail, and got this:

[vosque@vac ~]$ uptime
12:16AM up 4 days, 11:51, 5 users, load averages: 86.51, 53.51, 25.42

I tried to run ‘top’, and didn’t get that far. I had the NOC reset the machine, and everything appears well. Does anyone have any idea what was going on to cause such a high load?

Otherwise, if you see behavior like this, let me know so we can track it down.

Other Updates:

The Mission Control page has been updated to support and report on a lot more, so if you’re interested in the inner workings of the server, that’ll be a useful place to go.

Indra has been down for a while now. I checked the server out late last week, and the EXT3 filesystem decided that something was wrong with the disk, and went read only. Bad things happened. I tried to reboot the machine, but it went into maintenance mode (I’m guessing) and the vac < -> indra console cable appears to be disconnected. When I’ve got time to go up to the colo and fix it, I will. Probably not earlier than Friday, and not later than Wednesday.

New Server:

After making the offhand comment about getting an Xserve for, a couple folks have mentioned that they’d be willing to contribute to a new server fund.

I’ve been hesitant to accept money for in the past, since I keep the server running in my spare time for my own reasons, but donations for hardware are a different thing. If this idea has merit for you, please let me know that you’re interested, and what sort of hardware you’d like to see running. After that, we’ll see what comes of it.

Post Update Stuff

Alright, as most of you probably know, last week was down, in one form or another, for quite some time. As far as I know, no mail was lost or misdelivered, but that’s not a guarantee, so check your mailing list subscriptions. If you haven’t received some mail from a list in a while, that may be the case.

As of right now, the last network-facing service problem has been fixed: The old Symposium email system is now fully functional. The problem there was in the IMAP/POP3 login service– not in mail delivery. All Symposium mail was delivered at the same times that the rest of the mail was, you guys just couldn’t get at it. That’s now fixed.

We are now in OpenBSD 3.9, and it’s running a lot nicer than it used to. I also applied some fixes that may get rid of that random freezes we experienced in OpenBSD 3.7 and 3.8, but that’s sort of a blind hope at this point.

We’re now running MySQL 5, Apache 2.2, and PHP 5, so the most recent web crap should now be available for your usage.

I apologize for taking so long to get the upgrade through– I was actually moving the weekend this occurred, and the OpenBSD upgrade process is not friendly for those of us running OpenBSD 3.8 and using an Adaptec raid card in their servers. The official OpenBSD solution to the problem so far has been “buy a new server that is more politically aligned with open source”, and as soon as someone comes along and gives me a boatload of money, I’ll get right on that.

Hell, if someone handed me a boatload of money for a new server, I’d probably end up getting an Xserve anyway.

If you have any questions about the upgrade process, or you want some missing functionality that we missed, email me or admin at our shared domain.