Hands-On Testing and Analysis

Visiting Intel with SFD 12

Lotus Lightnes

Intel is always an interesting stop on the Storage Field Day whirlwind tour of Silicon Valley. During SFD8 we learned about how Intel tested their SSDs for radiation performance just up the mountain from DeepStorage Labs at Los Alamos and gave my podcasting partner Ray Lucchesi the nickname Cosmic Ray. With Intel at the heart of new technologies like NVMe and of course the elusive 3D-XPoint I was expecting a deep dive into low-level storage technology, and at SFD12 I was not disappointed.

The truth is that Intel, like no other company, determines the technologies we use in our data centers. Now that 99 44/100ths% of the servers we buy use not only Xeon processors but also Intel chipsets; the folks at Intel control the type and amount of memory and the PCIe lanes that provide I/O. That I/O becomes even more important as those Xeon servers become the controllers in an ever greater number of storage systems.

The good news is that Intel takes the responsibility that comes from dictating the computing architecture of the near future seriously by building open source projects like their Storage Performance Developer’s Kit. The Intel SPDK provides a library of drivers and functions that provide minimal latency access to NVMe devices for local applications and across the network via NVMe over Fabrics.

SPDK runs in user-space, polling the devices it controls avoiding the user-space to kernel context switch that adds latency to standard Linux I/O. Even with the SPDK limited to just one core, an SPDK system managed 1.6 million IOPS at under 300 microseconds latency accessing a set of P3700 SSDs. As we hit this point in the discussion, it was interesting to realize that the limiting factor in storage system performance may shift in the near future from the back end media to the PCIe and memory buses.

Today’s Xeon processors offer 40 lanes, for a total of 80 in a dual socket system. After the chipset, Ethernet, video card and other overhead there may be 64 lanes available to service 16 4-lane or 32 2-lane SSDs. Big storage systems are going to have to be scale-out or use PCIe switches that will limit their throughput to less than the sum of the SSDs.

With SPDK a prospective software-defined storage vendor has NVMe drivers and a standardized block layer at the back end plus iSCSI and NVMe over Fabric targets to manage the front end protocols. They can concentrate their efforts on value added services from basic data protection to encryption and deduplication and avoid reinventing the wheel without the performance.

Towards the end of his talk, we even got Jonathan Stern to endorse Fibre Channel over Token Ring, a technology invented over a sleep deprived break at Tech Field Day 3 by Ethan Banks (the father of FCoTR), Jason Bache, Bob Plankers, and this not so humble reporter among others.

Unfortunately, since their first 3D XPoint product wasn’t officially announced, the Intel folks could only hint at things we had all seen, since a copy of the spec sheet for the P4800X Optane (Intel’s brand name for 3D XPoint SSD) that leaked on the internet a week before. This lead to a bit of joking, but we were promised an Optane deep dive for the next Storage Field Day.

The good news is that the P4800X’s formal announcement was just a few days later. The first model has 375GB of XPoint with half a million IOPS at 10 microseconds latency with 30 drive writes a day endurance. It’s small as enterprise SSDs go nowadays and while I haven’t been able to get even a rough cost from anyone the P4800X is going to be expensive. I’d be surprised if it were less than $7 or even $8/GB or 5 times more expensive than Intel’s P3700NVMe flash SSD.

While it’s nice to see XPoint finally come to market, the real revolution starts with the next generation of Xeon servers that can handle 3D XPoint DIMMs and address XPoint as memory.

As always my travel expenses to Storage Field Day were paid by Stephen Foskett’s Gestalt IT and therefore indirectly by the sponsors including Intel. While Intel has been generous with SSDs and Ethernet cards for the lab over the years they didn’t give me anything more than a nice lunch at Storage Field Day.