Parallel NFS (pNFS): the War Is Close

Every Unix of Linux system administrator knows NFS, the Network File System, invented back in 1984. Its notable feature is a very simple syntax:

mount server.example.com:/exported/tree /example/mountpoint

This will mount the server’s directory “/exported/tree” on a local machine, making it accessible by the path “/example/mountpoint”. The server — server.example.com — also has to be configured, but it is not difficult, either. For many years, this setup was enough for many smaller cluster supercomputers, and it is still in use now (some parameters can be tweaked to make it more stable and faster). However, for the bigger machines, this presents a bottleneck, because there is only one server, and the whole system is limited by its performance.

Several workarounds were adopted over the years, finally resulting in truly parallel file systems such as Lustre and Panasas (which are not so different, by the way). However, Lustre is difficult to set up and maintain, while Panasas is easier to use and has more functionality, but is a proprietary solution resulting in vendor lock-in.

On the other hand, both solutions can provide tremendous speed: for example, Lustre was demonstrated to provide a 1,3 terabytes per second bandwidth for the IBM Sequoia supercomputer. The question is: can we have high speed, easy syntax and open source all together? The answer is parallel NFS (pNFS).

pNFS is the subset of the NFS version 4.1 standard, RFC5661 (see section 12). It defines the ways for clients to directly communicate with storage devices to read or write file data. This is the same idea as behind Lustre, Panasas and other solutions. What is different, however, is that usage syntax stays mostly the same as with the old good NFS, and the standard is open, so anyone can write an implementation.

There have been lots of work done in the recent years to finally get pNFS server and client code into the Linux kernel, but it appears it is not ready for the production use yet. They say the pNFS client is available in the recently released Red Hat Enterprise Linux 6.4 (February 2013), but let’s wait until more experience from production use becomes available.

However, it is clear that sooner or later some company or group will finally implement the standard — both the server and the client code — in a completely open-source way, and what happens next? This “what happens next” is a threat for some companies like Panasas. Indeed, one still can buy neatly designed Panasas storage hardware and run pNFS on top of it; but if you are not locked with that vendor, then nothing precludes you from buying competitors’ storage hardware (NetApp, EMC), and running pNFS on top of that.

So, Panasas basically recognised this trend very early, and acted according to the motto: “If you cannot oppose the movement, become its leader”. For Panasas, pNFS is a disruptive innovation. They have been investing considerable resources into development of pNFS to become one of its leaders.

Interesting times are ahead, and the storage landscape is going to change when pNFS reaches the wider audience, especially generic enterprise users, and not just HPC environments.

I also wonder what will happen to Lustre when pNFS becomes easy to use (i.e., becomes integrated into major GNU/Linux distributions). Lustre had a chance to gain widespread acceptance; will it continue to conquer that position? Or will Lustre remain a niche solution for extreme performance needs? And the final question: will we ever have a pNFS client for MS Windows?

To learn more about pNFS, check these web sites:

Open-pNFS.org, maintained by Tonian
pNFS.com, maintained by Panasas

(UPDATE from November 2013: It turns out that even the latest NFS version 4.1 standard still doesn’t provide enough mechanisms for distributed cache coherency. This means that multiple concurrent writes into the same file may lead to silent data corruption. Read the full story here).