Yeah, we'll just take that first network handshake. What could possibly go wrong?

Who, Me? Brickbats and bouquets are the way of things in the world of IT. Consider today's Who, Me? entry where our hero nearly fell on his sword when a bug bounty might have been more appropriate.

Our story goes back to the mid 2000s, when "Bill" (not his name) was working in the information security department of a large retailer.

"One of my many responsibilities was to implement a vulnerability management system," he said, "Everything was going fine: all testing of Windows and Unix systems had been successful, and we had placed it into production. The last step was to roll the system out to our AS/400s, which ran the guts of the business, in lieu of mainframes."

Bill explained the system to the AS/400 manager. He boasted of its success on the other platforms. He trumpeted its safety features. Of course the green light was given ("this was before we had implemented much of a change management system," he added).

The only proviso was that the job be run outside of business hours. Not a problem; Bill did the necessary scheduling and left for home.

Overnight all hell broke loose.

Upon his arrival the following morning, Bill was pounced upon by a gang of managers. "What did you do?" they demanded.

He'd simply scheduled a run of the vulnerability management system. What of it?

"Apparently most of the AS/400s had hung during the evening's production run," said Bill, "Orders had not been processed; a full-scale outage was declared; and the AS/400 team along with most of the IT managers and their bosses were on a crisis call for hours."

Bill saw his job and perhaps his IT career pass before his eyes. He protested: "But [the AS/400 manager] agreed to the run. And why didn't you call me? I could have stopped everything in five minutes."

True. However, the manager hadn't expected Bill to hit all the production systems at once ("good point," he admitted, "in a bit of hubris I hadn't considered that").

The connection between Bill's system and the outage wasn't made until the run was completed and systems restarted.

Preparing his resignation, Bill mumbled something about the software being actually written to minimize system load and none of the Windows and Unix systems had been touched. His excuse struck a chord with the AS/400 manager, who did some more digging.

It transpired that a vendor had also deployed some new software on the AS/400s. The outage, it turned out, had been a hang triggered by a combination of that new code and Bill's system. Nothing should be that flaky, and so a call was scheduled with the developer to work out what had gone wrong.

"The call was instructive," understated Bill. The supposedly skilled developer had implemented their system to take the first network handshake as the amount of virtual memory to allocate.

Not… ideal. And a HELO message sent by the vulnerability management system on the appropriate port, when converted to a number, might demand terabytes. The AS/400 would then frantically try and allocate enough memory, stopping any useful work from happening in the process.

We imagine there was somewhat of an intake of breath before Bill asked how an inappropriate packet would be dealt with. "They (rather smugly, I thought) replied that we should not allow that."

Suppose there was an accident? They couldn't be bothered to deal with that either.

• Lapping the computer room in record time until the inevitable happens
• Pop quiz: The network team didn't make your change. The server is in a locked room. What do you do?
• An international incident or just some finger trouble at the console?
• When civilisation ends, a Xenix box will be running a long-forgotten job somewhere

We'd argue that Bill's vulnerability management system had inadvertently exposed a gaping hole in the system. His management, not impressed with either the skills or attitude of the vendor, agreed. A call to the senior partner of the vendor was placed and a new version of the software was swiftly hustled up.

"And a great deal of cautious testing later, we were scanning again (though we permanently avoided touching the IP port that the software ran on)."

And Bill kept his job.

The Somebody Else's Problem Field was strong with this vendor. Ever accidentally stumbled over a flaw and assumed it must be your fault? Or did you leave a whoopsie in the code and thought "Nobody will ever come across this"? The kindly Register vultures await your confession. ®