TCP is a wire-centric protocol being forced to cut the cord, painfully

The venerable Transmission Control Protocol (TCP) is one of the foundation protocols of the Internet, but it's not so hot at mobile environments, says Juho Snellman of Swiss telco software concern Teclo.

The Register took an interest in Teclo's work after spotting this presentation to a SIGCOMM workshop.

In an interview with The Register, Snellman said the problem with TCP is simple: it was designed for fixed line environments, which are already in the minority in terms of client connections.

Since you can't dispense with the protocol, other fixes are needed – and that's where Snellman's interest arises.

“Mobile networks are extremely unpredictable and volatile,” he explained. To optimise TCP for the mobile environment, both the server side and the air interface have to be considered: “designing this, we had to shield the server from the radio network, and do smarter things with the radio network as well,” he said.

However, you can't just throw a “middlebox” onto the network to cache the traffic and hope it works.

“TCP is the only optimisation tool available for mobile”, he explained, since universal encryption makes caching and compression obsolete.

“All you have left is Layer 3-4 optimisation,” he said.

The biggest issue in trying to ship something into carrier networks is, naturally, performance. Teclo reckons it's got that sorted, with software that can handle 20 Gbps of optimisation on 10 million connections on a 2U node running on standard hardware.

The presentation says the hardware was Xeon CPUs 82580 or 82599 NICs, with optical bypass on the NICs to provide failover.

Snellman told The Register's networking desk the most important architectural part of the software comes from writing the networking to run in userspace.

“If you're using the operating system TCP stack and you want a million or 10 million connections, it's unpleasant”, he said.

The tendency for the kernel to accumulate excess code doesn't help: “In the kernel, for every new networking characteristic, someone adds more stuff – the data path becomes very complicated.”

An alternative at the kernel level was to try and write a new kernel module, but that's “a lot of work and you might run into licensing issues”, so the developers at Teclo wrote their software as a userspace application talking directly to the hardware.

(Snellman noted that there's work going on in the Linux community to address the kernel's networking issues, as a response to the rise of 40 Gbps and 100 Gbps Ethernet.)

As Snellman's presentation notes explain, the resulting architecture looks a bit like this:

This only needs around 1,000 lines of code, even including zero-copy “even for packets that we buffer for arbitrary amounts of time”.

And at the same time, so as to be transparent to the sender and receiver, the software has to maintain TCP options and TCP sequence numbers. That way, if the host hangs and the traffic switches to the failover, the session doesn't get dropped.

Keeping things transparent turned up another oddity, he said: the software “needs to be transparent to the TCP TTL fields, because some network nodes that use these to detect tethering.”

“Any time you're not transparent to some property, it will bite you.”

One of the surprises in trying to mobile-optimise TCP came from the amount of packet-ordering issues that arise in the networks, which Snellman said was unexpected given the horsepower carriers devote to shipping packets around.

“There is a lot of very specific reordering in there,” he said, “for example, tiny packets getting reordered ahead of large packets.”

One particularly bad result he cited was 30 segments in a mere 50 ms, and as the presentation notes, “reordering is poison for TCP”.

“That's very hard to distinguish from packet loss,” he explained. “If a packet gets ahead of 30 other packets in the queue, the host thinks the connection has lost everything and needs it to be resent”.

The answer is to develop heuristics to detect re-ordering, he said. ®