Nothing's working, and I've checked everything, so it must be YOUR fault

On Call The customer is always right. Except when they're not. Here we have a story from the On Call archives concerning connectors, telephones, and a user blessed with a little too much confidence.

Today's tale takes us back to the 1990s and comes from a former employee of a now-defunct telecom equipment vendor. We'll call him Felix, for that is not his name.

"I got to do a lot of on-site troubleshooting for the specific subsystem of the telecom kit that I had expertise in," Felix explained. One of the UK's telephone operators of the era was going through a period of modernisation and replacing decade-old kit with something decidedly more modern.

The modern devices were considerably faster and less power hungry than the elderly hardware. However, the rollout also necessitated some serious reconfiguration of the older gear. "A particular blade in the old kit," Felix told us, "could be made to operate in either an active or passive state depending on the orientation of a plug-in connector block.

"To change the state all you had to do was pull the blade, change the connector block orientation and plug the blade back in."

The hardware was pretty fundamental when it came to keeping the UK PSTN network going and a failure could mean that some end users might find themselves with dead landline phones. This being the era before the dominance of the mobile phone, "not having landline service was a VeryBadThing™," said Felix.

The call came in at 4:30pm on Friday, because they always do. It came directly from the tech lead of the operator's modernisation programme. Felix recalled the very much one-way conversation.

"It's all gone TITSUP!" Felix recalled him screaming down the phone, "Location X is failing to sync with the rest of the network, I've checked all the hardware and the config and it should work. But it's not!"

All we know about Location X was that it was a very, very important customer and it being down was costing the telephone operator an unthinkable sum per hour in terms of lost revenue as well as reputation.

Felix adopted the diligent approach familiar to all those who take such calls: "Has anyone changed anything? Are you absolutely sure the config is correct?"

"Of course it is!" bellowed the caller, "and we need someone down here to fix it ASAP!"

"Of course they did," sighed Felix.

• You've stolen the antiglare shield on that monitor you've fixed – they say the screen is completely unreadable now
• Who you gonna call? Premium numbers, but a not-so-premium service
• The Filth Filter is part of the chipset, honest. Goes between the TPM and SEP. No, really
• What came first? The chicken, the egg, or the bodge to make everything work?

Being possessed of the shortest of straws, it fell to Felix to source a company car and all the equipment required to diagnose what had befallen the gear before undertaking the 250-mile (402km) trip to site.

"By the time I was ready to start work it must've been close to midnight," he said.

But at least he was there.

He checked the clock signals coming out of the kit and confirmed that, yes, it was definitely not synching with the rest of the network. Next, he looked at the hardware. Out came the blade and…

"Lo and behold the connector block was in the 'passive' orientation."

"If ever I've seen the face of someone who wanted the ground to swallow him up," said Felix, "Mr Tech Lead was him."

Wordlessly, Felix moved the connector block to the correct orientation and replaced the blade. Everything sprang into life. There had been no need for all the expensive test equipment nor, to be honest, his presence. He hadn't even had time to book a hotel in the panic to get to site.

"So," he surmised, "basically I travelled 250 miles on a Friday evening, with a car full of tens of thousands of pounds worth of test gear, under threat of 'the world is ending' only to unplug a plug-in connector, turn it through 180 degrees and plug it back in again...

"Back then I could claim call-out and overtime, though, so at least it didn't seem quite so bad."

As for who had done the deed, Felix reckoned it was more likely a local tech being helpful. It wasn't until the upgrade programme had got under way and other blades had been changed out that the problem reared its head.

Still, we've all had that call from the customer who was convinced everything was correct until a quick eyeball confirmed it most definitely wasn't. Did your connector cockup involve a blue flash or a red face? Tell all with an email to On Call. ®