How I broke a server 1000km away with heatgun and grinder

Keep the fun and games in here for now please :-)
Post Reply
AncientGeek
QFP80 - Contributor
Posts: 67
Joined: Thu Jan 12, 2012 12:54 pm
Location: South Shithole

How I broke a server 1000km away with heatgun and grinder

Post by AncientGeek »

Sunday. I've got a remote server to upgrade. Its in a rural area 1000km away.

So I use vpn to ssh into the server, do some checks and start the linux version upgrade. I do not bother to run screen first, mistake #1.

While that is happening I wander into the workshop. Some wires to solder , and afterwards insulate with heatshrink. I hang the heat gun up without disconnecting it. Mistake #2

Now I'm working on the wire wheel of the bench grinder derusting some old parts. The grinder is about 200 years old and shakes the bench. This shakes something small off the top shelf, which falls on something else which swings and knocks something else, etc, etc. Like the Incredible Machine.

The end result is that the heat gun falls and places its power cable neatly on the grindstone. Flash! Bang!

Darkness. Kids screaming.

Oh f***!

Flip the earth leakage back on after pulling the plug on the heatgun. So now I go to check my upgrade. Oh oh. It was at some critical point waiting for input from me. Can't restart the upgrade because it's already running. I kill those processes and try to restart the upgrade. Nope, dependency problems. So have to fix it manually, in the process killing ldap, buggering the authentication. Some backdoors and lots of swearing late into the night, its running, new kernel installed. I reboot.

Will it come up again? I can't take this! I look away and discover I'm holding a hipflask. Where did that come from?

mmm... scotch.

Yes! I can log in! Undoing my epic fail is complete.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: How I broke a server 1000km away with heatgun and grinde

Post by Fred »

Awesome story! :-) What sort of time frames were we talking about?
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
AncientGeek
QFP80 - Contributor
Posts: 67
Joined: Thu Jan 12, 2012 12:54 pm
Location: South Shithole

Re: How I broke a server 1000km away with heatgun and grinde

Post by AncientGeek »

I think I started about 10 am and ended about midnight.

Somewhere during that time I was also trying bleed brakes and the master cylinder collapsed completely. It all seems a bit hazy now.
Post Reply