Congestion or a Christmas cock-up? A Register reader throws himself under the bus

Who, Me? Let's take a step back in time for today's Who, Me? with a trip to the dying days of manual credit card imprinting.

John, our latest confessor, was working on one of the UK's very first Electronic Funds Transfer at Point Of Sale systems (EFTPOS) during the early 1990s. His customer was a then-major electrical retailer, with a dizzying 500-plus branches across the country.

Automatic credit card authorisation was part of the deal as the existing system was hopelessly manual. Imprinting machines were used to record details, and transactions over a certain amount required a telephone call.

The wait for authorisation was not popular.

• Accidentally wiped an app's directory? Hey, just play the 'unscheduled maintenance' card. Now you're a hero
• Don't cross the team tasked with policing the surfing habits of California's teens
• Vegas, baby! A Register reader gambles his software will beat the manual system
• Housekeeping and kernel upgrades do not always make for happy bedfellows
• DBA heroes don't always wear capes. Sometimes they just have a bunch of forgotten permissions
• You would expect a qualified electrician to wire a building to spec, right? Trust... but verify

"Also," added John, "each branch received a daily list of known fraudulent cards that staff were supposed to check against; but the list was always out-of date."

The whizzy new EFTPOS system, running on the mighty OS/2, could do all this electronically via a X.25 network. The person facing the customer would get an immediate response and not even have to thumb through the naughty card number list.

John pulled the relevant data specifications from his local library and had the card authorisation component up, running and validated ahead of an October roll-out and code-freeze. No further software changes would be permitted until after the January sales. After all, everything had been tested, right? What could go wrong?

But something did go wrong.

As December arrived, John noticed a small but growing number of failing transactions. "Random authorization sessions were dropped," he said, "it looked like they were timing out."

"The obvious answer was that the problem was down to increased network congestion as Christmas approached."

And that should have been that. It was clearly the network to blame. Probably some idiot in the back office...

John decided that perhaps he should do a bit more investigation, just in case, before directing the pointy finger of blame anywhere.

And, as it transpired, that finger would have needed to have done a 180 degree turn back at John.

"I found that the affected transactions were consistently failing after 300 milliseconds. But according to the spec, the timeout should be 3 seconds."

Rather than 3,000 milliseconds, John had set a timeout of 300 milliseconds in his code. Nearly 30 years on, he admitted to being a little hazy about the exact number, but suffice to say he was out by a very, very long way.

"Far from being inadequate, the network had been fast enough to process all the transactions in a tenth of the allowed time."

There is an acronym in IT: PICNIC – Problem In Chair Not In Computer. And it was John in the chair.

With nobody else to blame for his mistake, John owned up. And was very surprised at the reaction of the team he'd planned to throw under the bus if he'd gone with his first instincts.

"The retailer's network guys were ecstatic about how fast their network was, and convinced the suits that the incorrect timeout was an honest mistake, couldn't have been foreseen etc.

"Even better, they were fine with deploying a patch that included the proper timeout, despite the fact that it was during the pre-Christmas lockdown."

John's lesson from the experience? "Sometimes not blaming the 'obvious' culprit, even if it means taking the blame yourself, is the right thing to do."

Ever found yourself wrong by a huge margin and confessed your error? Or did you cast an innocent party under the wheels of the bus to save your own skin? Time to confess with an email to Who, Me?