Back to the good old days when I worked as a system administrator, I remember that I wanted to implement a Lustre file system for the two small clusters that we had (64 and 20 nodes, respectively).
However, after learning that it requires Linux kernel patches, seeing how many people ask for help on the mailing list, and finding a 500-page manuscript on installation and basic management, I decided to wait. Seems that Lustre was mostly for big corporations at that time. Besides, it was not compatible with Microsoft Windows-based clients which was a crucial requirement for us.
Nevertheless, the installation and management difficulties are likely to diminish with the advent of Chroma, the software by Whamcloud.
In the beginning of 2010, Oracle acquired Sun Microsystems, and there came a time of uncertainty. The future of Lustre was unclear, and the community didn’t know who will take care of its future development. In the end of 2010, Oracle ceased to develop Lustre. Happily, by that time several companies and alliances have emerged, including Whamcloud who employed several senior Lustre developers.
Now Whamcloud seems to be the company that puts the biggest effort into Lustre development. Their product, Chroma, announced in 2011, is essentially a software that provisions and manages Lustre installations through the Web GUI.
I am not sure if Whamcloud will turn Chroma into open-source software with time. After all, it was venture capital that backed up Whamcloud, allowing us to enjoy new releases of Lustre. So, the investment has to pay off.
On the other side, when the community sees what Chroma can do, there could be efforts to create a comparable open-source software. There have previously been certain efforts, but due to the insufficient scope and certain lack of coordination they were not very successful.
Also interesting is that Chroma will help storage market players to build complete Lustre appliances. Finally we would be able to behold a rack of storage hardware that just needs to be shipped and put next to the other racks in your cluster. Several InfiniBand cables would connect this rack to the cluster fabric, and it would just work out of the box. That’s how it should ideally look like. And you would administer that system right from your browser.
Of course, this scenario doesn’t cover issues of Lustre clients — the compute nodes. In order for the nodes to (seamlessly) become Lustre clients, support from Linux distributions is essential. I believe this is the area where much can be done.
For those wishing more details, Dr. Joseph Landman provides an analysis of Lustre’s future (and promises to follow up) in the first part of his article on Sun Microsystems’ high performance computing assets that remain after acquisition by Oracle: “More than a year in, and where are they now?”
