or, "When there's no more room in Hell, the servers will walk the Earth."
Five years ago, Daily Kos was in dire straits. Booming traffic kept bringing the site to its knees. First DKos moved from a shared server with lots of sites to a server with only a few higher traffic sites. That failed to be enough, though, so after flirting with Slash, Markos raised $2500 (surpassing the original goal of $1000) to purchase a dedicated server to run the site and handle the transition to Scoop.
On Thursday, that server was shut down for the last time.
The writing had been on the wall for quite some time. Every once in a while, swish-e processes would eat up all the available memory and CPU and the server couldn't recover, since while one process eating all the memory and CPU would be killed, another would be waiting to take its place. Going into a death spiral, the server would cease to respond and have to be power cycled to recover.
There were other fears with it as well. If anything broke on it, replacement parts would be difficult to come by. Expansion was a non-starter. Finally, it was stuck running FreeBSD 4.11 and could not be upgraded any further because the 88MB root partition simply did not have enough space to permit trying to upgrade across major revisions.
Once, it ran all of Daily Kos. As time went by, its role shrank until it just ran outgoing mail for the site and jotter's search stuff. Despite its age, years of accumulated cruft, and struggles with its workload, it soldiered on for many years.
Plans had been in place to move mail and search to a new server for a while, but such a major move requires planning and thought. On Wednesday, all those plans went out the window. The old server froze up again for the second time in two days, so I went and hard rebooted it since nothing else would bring it back in that situation. I waited and waited for it to come back up, but it never did. Growing a little concerned, I connected to its serial console and saw that it was waiting at a single user prompt - /var was corrupted. I ran fsck to fix the errors as suggested, but it was bad. Very bad. Scores of hard read errors scrolled by while I watched helpessly, the superblock was completely hosed, and the "lost+found" directory was gone. I was also given the impression by the folks at the data center that that server was making a loud racket while I was trying to resuscitate it. I ran it a second time, but all the same errors went passing by while I sat and waited for it to get fixed. It wanted me to fix the filesystem again, but I could tell it was never going to get properly fixed, so I tried rebooting. Miraculously, it came back. Why it managed to come back I have no idea, but it gave jotter and I the time we needed to get the important stuff off of it before it failed for good. It even kept running until I pulled the plug on it once the switch was complete, still working away.
Its service may not be over yet. It's going to be sent to me, and I'll perform surgery on it and try to bring it back to life. If it's only a busted hard drive, then it'll be back on its feet serving Daily Kos again in its retirement, working as a development box in its old age. That $2500 server from five years ago might not be done yet, but even if it is it's had a long career. Take a moment today and pour a forty out on the ground, or sacrifice a bull, for the computer that worked for you just as long as it could.