All day yesterday I was in a great mood. I'd decided to spin off the "PCtown" storyline into its own mini-run comic in order to resume the M-F publication schedule of Help Desk and it felt like things were going to return to normal.

Late afternoon I received a notice that Drupal, the CMS I use to publish Help Desk, had released an update that fixed a critical security flaw. So I downloaded the update, applied the update, and updated the database...

... and the site broke. My relatively good day turned into a pretty horrible evening... and a pretty lousy today, too.

At first I thought "Drupal's update has screwed up my site!" I even went so far as to post that on their forums -- but, for the record, that wasn't the case. There was something wrong with my database -- I still don't know what -- but it was triggered when I ran the database updating script.

I didn't panic. Instead, I took my most recent database backup, did a little tinkering to bring it up to date, and I moved it live. And lo! Everything seemed to work! It even automatically published the comic I'd set to show up at 3AM Thursday morning. I went to bed tired, but triumphant. I had, I thought, fixed the problem, preserved all my data, all my user comments, all my user accounts, and all their bookmarks.

Thursday morning around 10 CST I woke up and checked the site. It seemed fine. No comments on the comic, which was a minor blow to the ego, but it was still running. I create a site news post to explain what had happened the day before, hit the "Submit" button and... I was suddenly transported to a random site page. The post never showed up. The same with comments: I could create a comment, preview a comment, but as soon as I tried to submit a comment I was sent to a random site page.

It took quite a while to figure out what had happened (and I needed some help from one of the posters on the Drupal forums) but I eventually figured out that for some reason my backup database was missing an entire week of comics - September 1-5 - but one of the internal tracking tables for the database still expected the comics to be there... this meant that everything published after that date was off by a count of five, the database couldn't match up anything properly, and trying to add any data to the site created an interesting "visit random page" effect.

So I started going through other backups... same problem. In fact, I had to use a backup made in August in order to get something that worked... and then I had to backfill all the important stuff, like comics and a few site news posts. Ultimately I managed to piece everything together, and it's a great argument for why people who run websites need to back up their data. That said: if the data is bad to begin with, backing it up just means it's bad in more than one place. Which is why it's good to have a historical chain of backups. I'm glad I haven't done any backup pruning lately (though it's probably time to do that now since most of them are buggy.)

The downside of this is if you have created an account on this site, Eviscerati.net, or Eviscerati.org since August 14 -- or if you have used the bookmark feature after August 14 -- or if you have posted a comment to this site after August 14 -- all of that is now gone. My apologies to everyone inconvenienced by it... but be glad I'm not a bank!