Oracle? On my server? I must have been hacked! *Penny drops* Oh sh-

Who, Me? After a hopefully relaxing weekend, we at El Reg want to kick off your week the right way – with a full-scale facepalm.

And so we bring you this week's instalment of Who, Me?, where readers share their cock-ups, large and small.

This week, meet "Wallace", who wrote in to tell us about a time he forgot himself – almost literally – while working as a newbie sysadmin at a smallish software development house.

"This was in the early 2000s, just before the days of broadband," he said.

"Our internet connectivity was via a whopping 128K of ISDN and thus the company's website was hosted on a Colo-d physical server elsewhere in the city."

One afternoon, Wallace was connected to this machine by SSH and needed to reconfigure the network settings.

"I made my changes and, as root, naively typed 'network stop; network start'.

"The 'network stop' executed perfectly. It shut down the network and killed my SSH session, meaning the 'network start' never executed. A rookie error but easily fixable."

However, it seemed things were more serious than Wallace had hoped.

"I phoned up the hosting provider, gave the technician my password, and asked her to log in and restart the network service. Except the password didn't work. We tried again. No luck."

Wallace asked her to reboot the server and try the password again. "Computer still said no," our anti-hero recalled.

"Next I got her to use the old 'init=/bin/sh' trick and reset the password. This worked and she could log in and confirm network connectivity."

Except that, from his desk, he couldn't SSH or load his company's website. Wallace hopped in a cab out to the hosting provider's facility and was shown to the shelf hosting his kit.

"It was literally a shelf – full of tower cases with no locked cages or other niceties," he recalled.

"It was my first time seeing the hardware in question, as it had been procured and installed before I joined the company."

After logging in with the new password, Wallace started looking around and realised that things definitely weren't right.

"Files that had been there that morning had disappeared. Files I didn't recognise had been added," Wallace said, wondering whether he'd been hacked.

"If so, the hackers had seen fit to install a full version of Oracle too, which struck me as a little cruel and unusual, even by the low standards of your average cybercrook."

And then, the penny – slowly, so slowly – dropped. "I was working on the wrong machine."

The full horror dawned on him, as Wallace realised that "an innocent customer had had their server forcefully rebooted and their password changed, and I was now bumbling around in their filesystem".

Meanwhile, his actual server was on the next shelf along, patiently waiting for someone to log in and type "network start".

"I explained all this to the poor technician, made my excuses and left," he said.

"My last memory is of her looking somewhat shaken. Presumably she was wondering how she was going to explain all this to her boss and their customer."

A few months later and ADSL arrived in his company's area and Wallace was able to bring the web server back in-house.

"It was a relief to know that all my subsequent mistakes were now being inflicted on the correct piece of kit," said Wallace.

Have you ever figured out why something has gone wrong just that little bit too late? Tell Who, Me? and we will gladly share your woes with El Reg's readers. ®