Scanning an Exchange server for a virus that spreads via email? What could go wrong?

Who, Me? Just like clockwork, another weekend is over and Monday is here again. To lighten the load, El Reg is offering you the latest instalment of Who, Me?, our weekly sysadmin confessional column.

This time we meet "Romeo", who was working at a large music company in London at the time in question.

It was his first job for a big multinational and the firm had just been hit by the I Love You virus that crippled systems all over Europe back in 2000.

"All our systems were taken offline for two days to be scrubbed clean," Romeo told us. "I was tasked with cleaning the servers in our Spanish HQ remotely over the network."

It was a Friday, and with the boss out of the office, Romeo was hoping for an early finish so he could enjoy the capital's night-life. But, as readers will guess from the fact he's written in to Who, Me?, this wasn't to be.

"I had scanned and cleaned most of the servers during the day, and just had the mail server to go," he said. "I logged on remotely using RDP [Remote Desktop Protocol] using a full local administrator. I installed the antivirus update, shut down the exchange server, and started a scan."

Our would-be pub-goer watched passively as the scan sped through the directory structure of the C: drive, not finding anything, and started on the drive hosting the Exchange server mailbox database (EDB).

Now with the bitter benefit of hindsight – and a couple of decades' experience under his belt – Romeo could see what's coming. It wasn't quite the same back then.

"In retrospect, one would know that the EDB contains the users' email... and that a virus that spreads via email might also be stored in the EDB," he said.

And sure as eggs is eggs, "the scanner found thousands of copies of I-LOVE-YOU virus in the EDB".

Without prompt, it followed the antivirus configuration setting, which just happened to be set to delete.

When the scan was finished, Romeo was almost ready to down tools; he just had one job left, to restart Exchange services.

"When it refused to start, and the error – file not found – was in the logs, the blood drained from me, and all thoughts of having a nice relaxing weekend evaporated."

After hours of trying to recover the file, Romeo had to call a colleague in Spain and ask him to put the previous night's DLT backup into the drive.

"I restored the EDB, but the Services still wouldn't start," he said – and so his nightmare continued.

"Anyone who worked with Exchange 5.5 knows that fixing a corrupted or out-of-sync EDB is a dark art," he said.

"I ran every sequence of recovery commands I could find on Microsoft Technet, and somewhere close to breakfast on Saturday, the Exchange service finally started."

Romeo told us that losing all of Friday's emails for several thousand users, and causing a serious outage would normally have cost him his job.

Luckily for him, then, that there was a pre-packaged scapegoat in the form of a Europe-wide virus attack, with systems "up and down like yo-yos", and companies concerned about their security.

"I sent out an email to the Spanish Distribution List apologising for the outage, blaming I-LOVE-YOU, as a particularly aggressive email virus. And, as it had infected everyone's email, all of that day's email had to be deleted."

Romeo didn't just get away with it – he passed with flying colours.

"My boss thanked me for my diligence and hard work, and for staying throughout the night to make sure the business would have email as soon as possible."

He added that no one noticed or knew what had really happened – which means there may be some irritated users who lost important emails, to whom Romeo offered a heartfelt apology.

How have you got away with your IT cock-ups? Have you ever wiped out your company's emails – or worse? Tell Who, Me? about it in strictest confidence and we might publish your tale next week. ®