How I broke a server 1000km away with heatgun and grinder
Posted: Mon Apr 29, 2013 2:09 pm
Sunday. I've got a remote server to upgrade. Its in a rural area 1000km away.
So I use vpn to ssh into the server, do some checks and start the linux version upgrade. I do not bother to run screen first, mistake #1.
While that is happening I wander into the workshop. Some wires to solder , and afterwards insulate with heatshrink. I hang the heat gun up without disconnecting it. Mistake #2
Now I'm working on the wire wheel of the bench grinder derusting some old parts. The grinder is about 200 years old and shakes the bench. This shakes something small off the top shelf, which falls on something else which swings and knocks something else, etc, etc. Like the Incredible Machine.
The end result is that the heat gun falls and places its power cable neatly on the grindstone. Flash! Bang!
Darkness. Kids screaming.
Oh f***!
Flip the earth leakage back on after pulling the plug on the heatgun. So now I go to check my upgrade. Oh oh. It was at some critical point waiting for input from me. Can't restart the upgrade because it's already running. I kill those processes and try to restart the upgrade. Nope, dependency problems. So have to fix it manually, in the process killing ldap, buggering the authentication. Some backdoors and lots of swearing late into the night, its running, new kernel installed. I reboot.
Will it come up again? I can't take this! I look away and discover I'm holding a hipflask. Where did that come from?
mmm... scotch.
Yes! I can log in! Undoing my epic fail is complete.
So I use vpn to ssh into the server, do some checks and start the linux version upgrade. I do not bother to run screen first, mistake #1.
While that is happening I wander into the workshop. Some wires to solder , and afterwards insulate with heatshrink. I hang the heat gun up without disconnecting it. Mistake #2
Now I'm working on the wire wheel of the bench grinder derusting some old parts. The grinder is about 200 years old and shakes the bench. This shakes something small off the top shelf, which falls on something else which swings and knocks something else, etc, etc. Like the Incredible Machine.
The end result is that the heat gun falls and places its power cable neatly on the grindstone. Flash! Bang!
Darkness. Kids screaming.
Oh f***!
Flip the earth leakage back on after pulling the plug on the heatgun. So now I go to check my upgrade. Oh oh. It was at some critical point waiting for input from me. Can't restart the upgrade because it's already running. I kill those processes and try to restart the upgrade. Nope, dependency problems. So have to fix it manually, in the process killing ldap, buggering the authentication. Some backdoors and lots of swearing late into the night, its running, new kernel installed. I reboot.
Will it come up again? I can't take this! I look away and discover I'm holding a hipflask. Where did that come from?
mmm... scotch.
Yes! I can log in! Undoing my epic fail is complete.