Dog walkers, the San Andreas fault ... and the storage industry

Comment Mainstream storage arrays are being harassed by a wolf-pack of sleeker competitors. Looking at storage is like looking at a dog walker on the beach with a hundred howling dogs of all sizes and breeds on the leashes.

Some are puppies, some young and sleek, some old and fat.

They bark and growl and run in all directions constantly, some forwards, some backwards, some sideways, some refusing to move. The dog walker is tugged first one way, then another as the dogs follow scents, chase each other, and try to screw each other too.

Over time though, the dog walker moves steadily along the beach – and the walker is the center of gravity of the storage industry; the hounds are the suppliers. Where is the walker going? Are all the dogs going with him? Or are some, like the aloof high-end enterprise canines, staying put and not going with the rest?

There is a San Andreas fault line running up and down the IT industry. It separates servers from storage and there are rumbles as new forms of storage erupt like subterranean volcanoes, their lava welling up into the fault and pushing the storage tectonic plate away from the server plate. The hot edges of the server and storage worlds are where the action is at its most intense.

Tremors through the industry

It all began in the 1995–2005 period, when it became apparent to many people that the classic dual-controller and monolithic storage arrays were the problem, not the solution.

There were and are several anti-disk array forces and this is what they said:

• Separately sourcing servers, storage, and networking is complex. Disk arrays are slow, expensive in both CAPEX and OPEX terms, complex – and they ran clunky, old software. Converged systems, then hyper-converged systems, they said, replace them with our simpler and better combined server, SAN, and networking systems for private clouds.
• The public cloud people said “No, replace them with our Amazon, Google, or Azure cloud storage and transfer CAPEX spend to OPEX with huge savings on enterprise data center costs.”
• The flash array people said replace those slow hunks of spinning rust with our flash drives and get data in a flash instead of waiting an age for disk latency. Their purist vision: abandon disk – you know you must – and head toward an all-flash data center.
• Hybrid array vendors said use our new, lean-and-mean software with a mix of SSDs and disk to get a combination of flash speed and disk capacity that's faster, simpler, and cheaper than traditional arrays. And there won’t be enough flash to replace disk for a long time.
• The object storage people said file systems were flat out finished, and their tech was the only way to store, access, and secure billions, even trillions of files which trad' arrays and scale-out filers, handicapped by clunky RAID protection schemes, couldn't handle well.
• A little later the Big Data people said classic arrays were data choke points and you needed scale-out nodes with parallel access, each processing their own local data; the Hadoop grid idea, to get insights from bulk stores of unstructured data.
• The server flash people said to put flash as close to compute as possible and get rid of both disk latency and network latency at a stroke.
• Software-defined storage folks said array hardware lock-in was inflexible and costly. IT users in the land of the free could free themselves from storage hardware lock-in by using SW-defined storage.

That’s eight barbarian storage tribes assaulting the six mainstream array vendors; Dell, EMC, HDS, HP, IBM, and NetApp. They’re winning too, forcing change on the mainstream vendors and taking business from them.

There is little happening at the archive end of the data life cycle. Tape endures and big, fat, cheaper, shingled disks are making progress as the idea of a nearline or online archive takes off.