Cisco mostly silent on when and what it knew about malicious WebEx wipeout

Comment Cisco this week launched a version of WebEx Classrooms, a version of its online collaboration suite tweaked for educational purposes and promised to enable “secure hybrid learning”.

Assuming nobody deletes the Classrooms from Cisco’s cloud.

Which may sound like an outlandish thing to have happen, save for the fact that last week we learned that the September 2018 WebEx Teams outage was caused by a former employee who was able to access Cisco’s AWS account five months after leaving the networking giant and deleted chunks of its cloud infrastructure.

Anyone can make an infosec mistake.

But this was three mistakes: the staffer should not have been able to get in, the deletion of 450-plus VMs should have been stopped by tools like Cisco’s own performance monitoring product AppDynamics, and disaster recovery should have required hours, not weeks.

And Cisco is not just anyone.

Indeed, Cisco is not just the networking market leader, it’s also the networking market leader that increasingly suggests cloud-hosted tools are the best way to manage its products.

Which matters because the manufacturer has not handled this incident well.

Back in 2018, your humble hack asked Cisco about the WebEx teams outage, and the IT titan told me: “The service interruption was the result of an automated script running on our Webex Teams platform which deleted the virtual machines hosting the service.”

“This was a process issue, not a technical issue,” Cisco added at the time. “We continue to investigate the causes of the script being run, however we are confident that this is an isolated incident and processes are in place to prevent any recurrence.”

The words Cisco chose in 2018 could have been a faithful description of what it knew at the time. Well, it's now 2020 and I’ve not seen Cisco admit the real reason for the outage – deliberate sabotage – to customers. The world had to wait for a court case involving an ex-employee to emerge before being able to join the dots.

Because Cisco did not proactively disclose to customers what it now knows, nor when it knew it, I asked the company this month why it chose to inform users that automation-gone-wrong was the reason for the outage, why it maintained that line, how the employee retained valid AWS credentials for so long, and whether Cisco has since enacted measures to stop this sort of thing happening again.

Here’s what a Cisco spokesperson told me.

• “Our immediate focus in September 2018 was to address the issue as quickly as possible, ensure no customer information was compromised, and implement additional safeguards.”
• “We brought the issue to law enforcement and it was determined an individual used knowledge as a former employee to gain unauthorized access to protected systems and initiate the script that resulted in the disruption.”
• “This was an isolated incident and processes and safeguards were put in place to prevent any recurrence.”

To your humble hack’s mind, those are not particularly satisfactory answers because they don’t explain what Cisco knew, when it knew it, nor how it chose to communicate with customers. Maybe Cisco decided not to discuss the incident while it was under investigation, but it would not be the first company to proactively explain why it was silent for so long.

Cisco has probably tightened up its CloudOps since this incident and has certainly made security a prominent part of its pandemic-purchasing pitch for WebEx and the new WebEx classrooms.

But Switchzilla hasn’t said what they are, a position that contrasts strongly with the verbose incident explanations offered by major clouds and the often red-faced admissions of failure in its own field notices.

Which brings us to this new, won’t-somebody-think-of-the-children moment.

It’s clear that Cisco is thinking how to turn them into cash. But it is unclear if it has really thought hard about how to stop its Classrooms crashing out of existence. ®