Sysadmin left finger on power button for an hour to avert SAP outage

Who, me? Welcome to the seventh instalment of Who, me? The Register's new column in which readers share stories of the times they broke stuff without any help at all from users.

This week, meet "Jeremy" who back in 1999 scored his first "real" IT job "as part of a team sent out to run the IT at a big publisher."

Said team was working on Y2K-proofing the publisher, which among other things meant updating firmware on a big fleet of old Compaq ProLiant 1000 and 1500 servers.

"These servers had mechanisms to prevent people from shutting them down by accident," Jeremy told us, going on to explain "small plastic cages in which the power button was hidden and could only really be pushed with a screw driver or a ballpoint pen."

Jeremy and his colleagues made decent inroads into the server fleet, but on the 100th upgrade may have let their concentration waver a little. They got as far as Windows NT's "You can now power off your computer" dialog, at which point Jeremy asked his colleague to push the power button of said server.

Which he did … but on the wrong server. And not just any wrong server, but a server running SAP. In production, natch.

Intellisense was off and developer learned you can't code in Canadian

READ MORE

At this point the nature of that power button became critical, because its design meant that so long as it was not released, the juice would flow.

And thankfully Jeremy's mate had pressed the button, but not released it.

The next hour of their lives was interesting. Jeremy's co-worker was left unable to move while the IT team scrambled, "getting everybody to log off from SAP and getting approval for emergency downtime. Then waiting for the SAP admins to shut down the server gracefully, before J could finally release the switch."

Jeremy told Who, me? that his mate asked to be relieved, as he was in a bit of pain. Those requests were denied due to the risk of the power going off and also out of a desire to make the poor chap suffer for his error.

What have you switched off by mistake? Click here to write to Who, me? with your story and perhaps we'll give it a run on a future Monday. ®