Ubuntu 12.04 Woes
Heaven forbid upgrades work smoothly... Precise Pangolin came out yesterday and I naturally upgraded my server to it. Took a while to download and install everything and so far it's taken the same amount of time figuring out how things have broken. Here we go:
resolvconf
I'm not sure when it was added or upgraded but before I was used to dealing with resolv.conf manually. Not that I had a lot to do: DNS suffix and nameserver was all. With those two statements everything was working fine.
But the 12.04 upgrade destroyed the configuration and replaced it with resolvconf's automatic configuration. Which got it wrong. It kept the DNS suffix (which should be easy enough to determine given the hostname of the machine) but the nameserver it replaced with localhost. Why? I don't know. I read through documentation and didn't see why it was insisting on using 127.0.0.1 when both the interface configuration and the DHCP information told it to use something different.
Whatever. I don't care. My website and database are broken because of this so I destroyed the /etc/resolv.conf symlink and reconfigured it manually. Eventually I'll deal with resolvconf but not right now.
mysql-server
As a nice show of security they changed the my.cnf configuration so that mysqld would only bind to 127.0.0.1. Thanks for the thought. But that breaks my website because it (and other running services on the network) want to connect to it by name. And I don't mean "localhost". So my website, email, and a couple other applications stopped working. So thanks but no thanks.
upstart, apparmor, and mysql
Today I find out that mysql broke again. No email, no website. Did I mention that I installed modsecurity a while back but had to disable a handful of rules because error messages were my only recourse to troubleshooting problems? Mono doesn't like logging... well, anything, despite my pleas for simply hitting up the Apache error log, so when there's a problem the only option I have is to see what the "live" website says. And modsecurity has an obsession with substituting any error message with a simple 403: no 404s, no 500s, no 503s, nothing. So when it can't connect to MySQL it shows a yellow screen of death. Normally that's okay because I get email notifications of when that happens. But no database means no email either.
I actually only found out that email wasn't working because every couple days I have to have my phone manually refresh email. What I mean is that my email is set to use server-push so I get messages soon after they arrive. But between my Android phone and Courier MTA, pushing occasionally stops. Which doesn't translate to "and my phone falls back to manually retrieving email". Instead I get absolutely no email at all. And if I forget to manually refresh every couple days or so then I miss out on every single email. For example, I missed out on an invitation to a Guild Wars 2 beta test. Had no idea I received it because nobody told me. But that's a different problem.
So I go the computer and see that the mysql daemon isn't running at all. Okay... so I try to restart it. Nope. "Job failed to start", which is a very helpful error message and tells me something I would never have noticed by myself. I check one thing after another and get absolutely no help. Then I learn that mysqld starts via upstart now and eventually make my way to the upstart logs.
They tell me the AppArmor couldn't open some file somewhere. I look and sure enough, a file is missing. Just one. Half of the configuration is there, half of it isn't. Don't know what happened but I don't care. Do I regenerate the missing half of the configuration? Tried. File is completely missing. Not even in the mysql-server package. I don't know what should appear in there and do the next logical step: disable the other, working half so that AppArmor doesn't bother doing anything. Restart and mysql is working again.