Storage evolution led by virtualisation

Storage evolution led by virtualisation by Phil Reed.

Over the last decade there has been a dramatic change in storage technologies which has been fueled by the movement to virtualisation.

For years we had standalone servers with direct attached storage, each server one or two applications to ensure reliabilty and performance. Virtualisation brought server consolidation, reducing power costs and required rack space. It also brought the possibility of High Availiabilty and automatic performance balancing. These features required access to shared storage which led to the proliferation of SAN’s (Storage Area Networks) initially Fibre Channel based then iSCSI.

As more demanding applications became possible and brought the need for increased performance in I/O this led to larger SAN’s. Applications such as VDI (Virtual Desktop Infrastructure) and the specific issues that they have such as “boot storms” require an even higher I/O requirement. But this needed to come at a cheaper price so was throttling the VDI market.

The advent of flash storage led to hybrid storage arrays with tiered software management of data placement. Some All flash arrays used as front end cache to provide performance, expensive but cheaper than traditional costly expansion of SAN units. It also led to a demand to decouple storage performance from storage capacity.

IT profesionals begin to recognise that with SSD’s overcoming the data access speed barriers at the storage layer, moved the bottleneck to the network layer. Requirement for 10GB Ethernet or expensive InfiniBand connectivity has therefore grown and recently extended to 25GbE and 40GbE.

Then the Hyper-Converged solutions – Nutanix, EVO:Rail, Tintri – using mixed flash and traditional storage with closely coupled servers . The need for a cheaper solution but with performance was leading manufacturers to look at software based virtual SANS.

Next phase was local server side caching using SSD and then RAM only caching (PernixData and Infinio). These use a combination of caching and deduplication techniques to make best usage of spare RAM capacity to increase Read IO performance.

Software based virtual SAN utilise local direct attaached storeage to provide shared aceess to data using SSD as a cache and mirroring data across all servers for resilience. An evolution of storage which has brought us back to servers with direct attached storage (sort of).