Will We Ever See InfiniBand in Desktop Computers?

[Update: the post is not actually about InfiniBand as such, it’s more about convergence that can be brought by the new Intel Omni Scale Fabric :-) Please see the comment below the post, and thanks for your time!]

When I was a system administrator (and it was long ago!), I dreamed that all desktop computers in the world would use one single type of network technology. Of course, I wanted to minimise the amount of maintenance for myself — but it would benefit our users, too.

Workstations of our engineers were connected to the Gigabit Ethernet switch, and compute nodes of our clusters were connected with thick and inflexible black cables to the InfiniBand switch. Head nodes of clusters were connected to both switches and acted as “routers” between these two network technologies. Wouldn’t it be better if all computers would reside on one type of network — for example, InfiniBand, because it is so much faster than Gigabit Ethernet?

We even thought of switching our workstations to use InfiniBand, but those expensive black cables with large and unwieldy connectors eliminated all elegance of the idea. Simple unshielded twisted pair (UTP) cable used for our Gigabit Ethernet network seemed a much better alternative: it was thin, easy to handle, and we could terminate the ends of the cable with simple tools. One day I read a press release (I guess, from Mellanox) where they speculated about an opportunity to run InfiniBand over twisted pair, albeit at a much lower speed. Unfortunately, this never happened. To use InfiniBand in your desktop computer or a workstation, you still need to buy an expensive InfiniBand network adapter, such as one below. Luckily, the newer adapters from Mellanox can speak both 40 Gbps Ethernet and 56 Gbps FDR InfiniBand:

Single-port and dual-port InfiniBand adapters from Mellanox. New models can speak either FDR InfiniBand at 56 Gbps or 40 Gbit Ethernet (auto-sensing or user-selectable). Note that in the real life the beautiful chip is hidden below the heatsink. Image source: Mellanox.

But Gigabit Ethernet hardware has for many, many years been available on desktop motherboards and even on notebooks! (I still own a 2007 Acer notebook with a Gigabit Ethernet adapter from Broadcom). Is it possible to produce an InfiniBand chip so cheap (pun intended) that it could be placed on every desktop or server motherboard? It could run at 10 Gbps speed over twisted pair, perhaps even sharing the same physical connector (what they colloquially call RJ45) with the on-board Gigabit Ethernet. This was the question I always wanted to ask the manufacturers — but didn’t have a chance to.

And then, at the recent International Supercomputing Conference (ISC’14) in Leipzig, Germany, I was standing at a table during the lunch time. Suddenly two gentlemen approached, with their badges clearly indicating they worked for Mellanox. I assumed this to be God’s intervention :-) So, instead of letting us all enjoy the lunch, I asked them a direct question: “Will we ever see InfiniBand in desktop computers?”

They looked at one another and seemed to be hesitating. I thought they were going to tell something like “Nobody would need such speeds in desktops”, so I made the first move and said: “This could be a slow-grade InfiniBand, maybe at 5 or 10 gigabits per second, running perhaps over twisted pair”. At this moment, they were not hesitating any more, and one of them replied: “No. We start at about 50 gigabits per second, and are aiming at one hundred. For desktops, there are companies that could make it cheaper and better than us”. My curiosity was satisfied, and quicker than I could imagine. It’s just the wrong market for them, the desktops. So, I switched topics (and no, I didn’t leave them alone with their lunch — that would be too simple) :-)

During the evening, I thought to myself — what could be that company that can bring fast connectivity to desktops (at least 10 Gpbs) and at the same time offer some convergence in network technologies between desktops and servers. And then there was a special talk by Rajeeb Hazra from Intel. And he talked about the Intel Omni Scale Fabric that will be integrated on the next Intel Xeon Phi “Knights Landing” chip.

Rajeeb Hazra from Intel delivers his special talk at ISC’14. Photo by: Tim Krieger, ISC Events.

I suddenly realised that this is the company that can provide high-speed connectivity to desktops. All they need is just to integrate the network adapter circuitry on the CPU die. I am sure they will do that in the not-so-distant future. And servers and desktops would finally co-exist on one network fabric, just being connected to it at different speeds.

For those of you rubbing their hands with excitement, there is something more to know: although Omni Scale is not InfiniBand, it will be software-compatible with it: “There is application compatibility between currently available Intel® True Scale Fabric and future Intel Omni Scale Fabric, so customers can transition to new fabric technology without change to their applications” (taken from the above-referenced press release).

(And yes, this whole story answers the question of what did Intel do for so many years with the network business assets it acquired from Fulcrum Microsystems, QLogic and Cray in 2011 and 2012).