Boss regrets pointing finger at chilled out techie who finished upgrade early

On-Call Friday is upon us once more, which can mean only one thing: it’s time for On Call, our weekly instalment of Reg readers’ tech support frustrations.

This week, “Luca” tells us how his hopes of a chilled-out Friday – and possibly plans to kick back reading this very column – evaporated in the face of an angry boss.

At the time in question, he was in charge of development and DevOps for a few clients’ websites, working in a medium-sized firm with 250 staffers across the country.

“When I started, the servers and services were quite a mess,” Luca told us. “There had been no upgrades for months, maybe years.”

Over the space of a few months, Luca was in charge of upgrading and patching everything that needed it, and moving the infrastructure to a cheaper, uncompromised and upgraded cloud solution.

“It took some time but all went well,” he said. “The websites were fixed, the load-time got to a 1.3s standard, which was a dream compared to the previous state.”

Even better, Luca finished a few days ahead of schedule, leaving time to check everything was working properly, and to see if there were any tweaks needed to improve the infrastructure further.

He was due to give a presentation at the start of the next week, but with the weekend approaching, Luca gave himself a big pat on the back.

“Friday comes and I was super relaxed about everything,” he said. “Feeling no stress at all, I got to work half an hour late, walking in slowly with a big cappuccino in my hand.”

But the scene that greeted him was anything but the serene peace we all wish for on POETS Day.

“The office was frantic,” Luca recalled.

“The moment I opened the door, everybody looked extremely nervous – then started blaming me, because not a single one of my projects was working. Thirty people had been waiting for half an hour for me to come in, so they could complain.”

Mastering the power of the understatement, Luca said his boss was “quite angry” because all of Luca’s clients had been on the phones making “friendly” calls about the problem with their sites.

With his reputation on the line, Luca asked for an hour’s reprieve to delve into the systems and figure out what was wrong.

“I tried loading the website... nothing. I tried SSHing in the infrastructure... nothing. I tried pinging the infrastructure... nothing at all,” Luca said.

Panic mounting, he began to think he was done for, as possibilities started racing through his mind.

"Did I get hacked? Did I forget or mis-configure anything? How is it possible that not a single server is reachable? WHAT HAVE I DONE?!"

The only option was a reboot from the web interface and to hope for the best, he said.

You haven't paid for X months, pay your bills to activate your account again.

But when he connected to the web interface, Luca immediately saw the problem – and that he wouldn’t have to take the flak for much longer.

Because, emblazoned on his screen was the alert: "You haven't paid for X months, pay your bills to activate your account again.”

And who was in charge of paying the bills? The boss, of course. “He had changed [his] card and forgot to change it on every single platform,” Luca said.

And, to make matters worse, the boss was filtering all incoming mail about invoices from that provider, and marking them all as read – so he didn’t notice a thing.

In the end, Luca said, his boss apologised in front of everyone in the office, “which felt amazing” – though he later took Destiny Child's sound advice and found another who could pay the bills, bills, bills.

Have you ever had the top dog grovel at your feet? Tell On-Call about the time you made the big man feel small by emailing us here. ®