The reluctant log trawler: The buck stops with the back-end

On Call Everyone's favourite day of the week, Friday, has waddled into view. Grab a steaming mug of schadenfreude and settle down with another tale from those Register readers saddled with the On Call phone.

Today's story comes from a reader the Regomiser 9000 has elected to call "Ed" and concerns the fun that comes from bodging around another person's bodgery.

Back in the late 2000s, Ed was working as a developer on IBM's iSeries and, for his sins, was also first line IT support.

His system formed the back-end of a website that allowed customers to buy and sell shares. A request to buy or sell was sent to brokers in real-time to get quotes. Customers then usually had 30 seconds to decline or accept the terms (and have their trade stamped with a unique "Guaranteed Price ID"). If no guaranteed quotes were forthcoming, the customer could also trade at somewhere close to the market price.

It is important stuff and tends to get auditors a bit feisty if there are issues.

Ed was happily tapping away at his keyboard when the first call came in: "Somehow, a trade was sent to the broker for stock A, but with the guaranteed price ID from a quote for stock B." The broker code did not validate for that and so the trade had been processed... "just for the wrong stock."

Could he possibly look at how it could have happened?

Peering at the database tables, Ed was able to see that the customer had quote requests for both stocks. Stock B had returned a guaranteed price ID, Stock A was market price. He could then see the trade for Stock A being sent with Stock B's ID. Not good.

Worse still, there was no obvious way the IDs could have been switched. Unless there was something funny happening with the web front end. Surely not?

He began trawling through the logs to work out the exact path the customer had taken. A thankless task, and one that only he had the knowledge to do (Ed told us he'd tried training others in the dark arts of log inspection, but his students tended to depart the company shortly after).

It wasn't long before he turned up some decidedly whiffy behaviour. Although quotes cannot be saved (just accepted or declined) it looked rather like the customer had multiple browser tabs open at once and was working with several orders at a time.

Surely the web front-end could handle that though? After all, pretty much all the major browser makers had a tabbed interface by then, so such behaviour was not unheard of.

It transpired that it couldn't. Ed was able to recreate the issue with a bit of faffing and multi-tab shenanigans: "If one quote had a data value and the other did not, the web app could end up filling in the blanks for one with data from the other. So the blank guaranteed price ID data item in one quote was replaced by the populated value from another quote," he told us.

Chuffed with his sleuthing, he submitted his findings and thought no more about it. The web team would probably have some work to do at some point, but the issue was not with his precious iSeries.

In our experience of the trading world, such a discovery could set off all manner of panic bombs. So rather than the peaceful weekend he was expecting, Ed received a surprise Saturday night call from the bosses.

"Apparently," he said, still sounding a little aggrieved, "they felt it was such a massive flaw that, despite this being the only occurrence in the system's 10-year life, it needed to be fixed before start of business on Monday."

Even worse (and Ed told us he still recalled the exact words a decade or so on) he was told: "We feel the cause of, and the solution to, this issue is on the iSeries."

The inability of the web app to handle multiple tabs and mangle customer data was not the problem. It apparently lay with the iSeries backend not validating for that particular type of borkage coming down the pipe.

Sunday was then spent on the fix which, Ed admitted, didn't take too long.

"What took the whole day," he grumbled, "was bodging the test broker trading interface to produce quotes on a Sunday when every aspect of its design was set up to make it only work on weekdays."

Still, he pushed on through and got the fix live, although we fear he may have missed that week's Antiques Roadshow. "I was not thanked nearly enough..." he growled.

Ever found yourself doing some hurried hacking when the "unthinkable" happened? Or been called out to bodge your way around someone else's cock-up? Share you story of unexpected weekend working with an email to On Call. ®