Gits club GitHub code tub with record-breaking 1.35Tbps DDoS drub

What's purported to be the world's largest distributed denial of service attack to date – measuring 1.35Tbps – knocked GitHub offline for a few minutes yesterday.

The massive tsunami hit at 1721 UTC. During the assault, the popular code sharing website's admins noticed thousands of systems and devices slamming GitHub's web servers.

Inbound network traffic peaked at 1.35Tbps, or 126.9 million packets per second, we're told. The sheer volume of data overwhelmed GitHub's computers, causing them to stop responding to legit users, and effectively fall offline. At that point, GitHub turned to Akamai to filter out the malicious traffic, ending the attack's effect after five or so minutes.

"Given the increase in inbound transit bandwidth to over 100Gbps in one of our facilities, the decision was made to move traffic to Akamai, who could help provide additional edge network capacity," GitHub's site reliability engineering boss Sam Kottler explained in detail on Thursday.

"Routes reconverged in the next few minutes and access control lists mitigated the attack at their border. Monitoring of transit bandwidth levels and load balancer response codes indicated a full recovery at 1730 UTC."

There was a second, smaller attack at 1800 UTC, which peaked at 400Gbps but this was absorbed without the site taking another dive.

The massive attack wasn't the work of an enormous botnet of hacked gadgets throwing junk traffic at GitHub. No, whoever orchestrated the record-breaking assault used a well-known but little-used technique known as Memcrashing.

It is likely we're going to see copycat attacks on other sites, given that the GitHub outage was the result of a second Memcrashing assault we've seen just this week.

Memcrashing works by exploiting memcached database servers that have been left open to the public internet with no authentication requirements in place. There are tens of thousands of such vulnerable systems online at the moment, ready to be press-ganged into knocking other services off the 'net.

Here's how the amplification attack works: a miscreant sends a small database command to an open memcached server, and, in the UDP packet for that request, sets the source internet address as the victim server's. The memcached database fires back about 50,000 times the amount of data it received in the command – a 203-byte request results in a 100MB response – and, well, you can see where this is going.

A small number of computers, running memcached insecurely, can therefore be unwittingly turned into trebuchets lobbing huge boulders at unsuspecting services and websites.

As it turns out these kinds of attacks, while massive, aren't too hard to mitigate. You can try blocking UDP traffic from port 11211, which is used by default by memcached, at the border or upstream, as Akamai was able to.

Meanwhile, database and network admins should secure their memcached installations so they cannot be abused – such as by turning off UDP support or firewalling off the software. Programmers should also think again when writing code that uses UDP: ask yourself, can this be exploited to reflect large amounts of traffic from small requests? If yes, go back to the drawing board.

The developers of memcached have disabled UDP support by default in future versions of the software in response to the attacks.

"Because of memcached reflection capabilities, it is highly likely that this record attack will not be the biggest for long," Akamai warned in its report on the incident.

"Because of its ability to create such massive attacks, it is likely that attackers will adopt memcached reflection as a favorite tool rapidly. Additionally, as lists of usable reflectors are compiled by attackers, this attack method's impact has the potential to grow significantly." ®