Adventures in mineral oil cooling

Let’s say that you and your pals need to build a high performance cluster to run a number of HPC apps. It has to be fast. Damned fast. Fast enough to beat the very best student-built systems from seven other universities.

Your only limit is power: you’ll get 26 amps, no more. Your adversaries are going to pull out all the stops, bringing loads of CPUs, GPUs, SSDs and fast memory. What do you bring?

If you’re a student at the University of Texas Austin, you’ll think outside the box – literally – and come up with a solution that allows you to pack more cores into the 26 amp power envelope. For the Student Cluster Competition at SC11 in Seattle, the Longhorns surprised everyone with a cluster that uses mineral oil immersion as its sole cooling mechanism. Green Revolution provided one of their cooling units to the Texas team and helped them install it at the show.

Using immersion cooling allowed Team Texas to remove all of the fans on their nodes, resulting in a large enough power gain to fire up a couple of extra CPUs. They’re not exactly sure how much extra oomph it gave them (they hadn’t had a chance to do an air vs. liquid benchmark), but they estimate it’s between 5-15%.

So what’s this system like? How does it go together? What’s it like to deal with? The video gives us a good look at the Texas Deep Fried Cluster system. We get to watch them remove and replace nodes as they re-cable the system as a precaution before the beginning of the competition. It’s a pretty interesting process (pretty messy, too) and it requires a surprising amount of arm strength. Why? You’ll see in the video…

Watch Video