Alrighty, here's the deal: I have this forum running on one of my Linodes. The other day I went to upgrade a vulnerable package on the server and the upgrade failed. I tried your standard fixes, but the Linode became unresponsive and I couldn't do anything with it.
Whatever, that's fine, right? I have weekly backups being made. So I go to restore one of those backups. The restoration failed, and the server won't let me log in via SSH.
So I open up the web console: kernel panic.
I try booting into recovery mode, try to run
fsck, comes back normal. So I can mount the volumes, but I can't figure out what's wrong with it. Try to repair broken packages, thing still won't boot. At this point I've got clients that I have to do work for, so I have no choice but to leave the forums offline for a while.
Come back a two days later, still can't figure out what's wrong. At this point Linode's support is like "Hey, we've noticed you've had several failed backup restorations. Need help?" Well, that's convenient, but they're only able to do a bit. They link me to some documentation, but it's not much and it doesn't help me resolve the issue.
Anyways... I'm five days deep into this without the forum (or my personal Linode for that matter) being online. I say "screw it" and just spin up a new Linode. I use
scp from the new server to copy all the apache configs, databases, etc and clone the
/var/www/ volume from the old server. 👍️
Then I decides to just swap the IP addresses of the servers rather than update the DNS records because I figure it'd be more convenient. Well, for some reason it didn't work immediately and I couldn't log into either server or access them remotely.
Had more client work to do, so I had to again put the forum on the back burner.
Well, today I get word that I'm not supposed to do work for my main client unless they specifically ask me to during the whole COVID-19 shutdown... so I figure "well, I guess I can fix the forums now."
Glad we're back online, friends! I'm excited for things to get back to normal around here. I know I don't post here much, but I read most threads!
Hope you've all been well, are staying safe, and are washing your hands regularly!