Lustre and Panasas Are Not So Different

Posted on August 21, 2012 by Konstantin S. Solnushkin

As vendors try to showcase their products, there is often confusion regarding the difference between two similar products, such as PanFS from Panasas, Inc. and Lustre (from Sun, then Oracle, then Whamcloud, then Intel). But in this particular case, the difference is not that big — you will see for yourself.

Panasas was founded in 1999 by Dr. Garth A. Gibson and Dr. William Courtright. Garth A. Gibson is a co-inventor of RAID technology — no wonder that his company excels in storage products. Panasas provides appliances — hardware and software bundled together and fine-tuned for high performance. This has always been a “closed-source”, proprietary product, which translates into a vendor lock-in.

Lustre experienced a tumultuous history of corporate mergers and acquisitions. Finally, source code was published under the GNU GPL license, but only the chosen few would dare to look inside. Everyone else is happy with pre-patched Linux kernels. Most development efforts seem to be done by Whamcloud, because they have the necessary expertise. Recently, Whamcloud was acquired by Intel. Theoretically, Lustre is still open source, so anyone can contribute, but for practical purposes, you still need to ask a single company (formerly Whamcloud, and now Intel) if you want to have new features added.

Similarity between the Panasas solution and Lustre is best represented by this figure taken from the Lustre manual:

Basic configuration of a storage cluster with Lustre. Source: Lustre manual by Oracle.

Continue reading →

Posted in Miscellaneous | Tagged , , , | 5 Comments

Intel MIC, aka Xeon Phi, aka Doubtful Creativity

Posted on August 20, 2012 by Konstantin S. Solnushkin

They created a monster.

It contains enough cores to be a computer on its own — yet it needs to be plugged into a “real” computer, and acts only as an accelerator.

It’s cores are based on P54C architecture, introduced on the market in 1994 (by now, almost 20 years ago). A single P54C core contains 3,3 million transistors (for comparison, a single core in a 4-core Ivy Bridge CPU contains 100 times more transistors). So, the cores in Intel MIC are (much) more basic, but at the same time, hopefully, more power-efficient.

Intel Knights Ferry (prototype board of Intel MIC). Source: Intel

One of the main benefits (as marketed by Intel) is the ability to create software using the same tools you use for other Xeon products. Note this: not the backwards binary compatibility with your existing applications (doesn’t work — you need to recompile them), but only the ability to use your habitual compilers and libraries. Continue reading →

Posted in Reviews | Tagged , , , , , | Comments Off on Intel MIC, aka Xeon Phi, aka Doubtful Creativity

Marrying NVIDIA Tesla and InfiniBand?

Posted on July 18, 2012 by Konstantin S. Solnushkin

My friend is working on a research project dedicated to many-core architectures, such as NVIDIA’s GPUs or Intel’s Xeon Phi, that have lots of simple cores best suited to straightforward computations.

Sometimes those cores need to communicate with their neighbours that are located far away — perhaps on another accelerator board in the same compute node, or even in a different compute node. And the cores are often so simple that they are not well-suited to MPI communications.

NVIDIA Tesla GPU accelerator. If only these boards had their own CPU that could boot an operating system, how many of them could you pack into a standard 42U rack? Image by: Ray Sonho. Source: Wikipedia

Hence, the key idea of the project is to offload MPI communications to the CPU of the compute node. The research team is now working on the MPI implementation that would facilitate this offloading. Continue reading →

Posted in Ideas | Tagged , , , , , | 2 Comments

Electrochemical Power Supply and Water Cooling, In One Go

Posted on June 29, 2012 by Konstantin S. Solnushkin

I was searching the IBM website to find info on their cooling system, Aquasar, in the hope to find a better way to learn about it rather than just looking at their hardware.

And indeed, I found something new and exciting. Seems that IBM researchers were not sitting idle lately. Of course you heard already they were going to cool computer chips by passing water through microscopic channels inside the chips.

But this time they went one step further and proposed to supply power to the chips using electrolyte that would pass through those microscopic channels, at the same time taking away generated heat. A very clever move.

More details are available in this presentation by Dr. Bruno Michel, Manager Advanced Thermal Packaging, IBM Zurich Research Laboratory.

Dr. Bruno Michel, IBM

Dr. Bruno Michel, IBM Zurich Research Laboratory. Copyright: IBM. Source: Flickr

The presentation also touches upon photovoltaic systems. According to the slides, their efficiency is less than 30%, and the remaining solar energy is wasted as heat. A scheme is proposed to use that heat to desalinate water. Many countries would really appreciate this.

UPD: IBM published their timeline of water-cooling achievements, starting from 1966.

Posted in News | Tagged , , , | Comments Off on Electrochemical Power Supply and Water Cooling, In One Go

T-Platforms and Water Cooling

Posted on June 28, 2012 by Konstantin S. Solnushkin

For ISC’11 in Hamburg, T-Platforms brought to their exhibition booth a prototype of a water-cooled system. I think I remember those thick black rubber tubes (and hope that was not a dream).

(UPD: A Google search reveals that they were, indeed, working on a water-cooled system, to be installed in the Moscow State University. The prototype system called “TB2W”was developed, and it is described in this T-Platforms’ brochure. Probably that was the system I saw at ISC’11).

Image: TB2W prototype from the above-mentioned T-Platforms’ brochure

So I decided they are working on a product that would implement water cooling. But, much to my surprise, one year later, at ISC’12, they didn’t bring anything water-cooled with them. Their sales manager told me they currently ship traditional air-cooled systems, and have also produced a “personal supercomputer” — a replacement for Cray CX1 product that is no longer shipped by Cray. How bizarre, I thought.

Just a couple of days later I was strolling around the exhibition hall, and came to the IBM booth to take a look at their water-cooled iDataPlex servers they used for SuperMUC, as they were on display. I was having a conversation with IBM representative when I saw a T-Platforms employee coming up to the booth, accompanied by a lady. They came to the water-cooled server, and he pointed his finger somewhere to the copper tubes and told: “Look here! Let’s see, that’s how they made the cooling”.

I was really surprised and, actually, I was just stunned. The IBM representative didn’t understand why I suddenly stopped talking to him, so I had to explain that those visitors were his competitors from T-Platforms. “Really?”, he replied. I can’t say he was pleased to hear that, but unlike me, he at least stood firm on the ground.

Later I had a conversation with another man whose company had a booth at the exhibition. He commented on the situation in the following way: “Yes, we saw that T-Platforms guy near our booth — but we recognized him, and so we advised him to stay away”.

Posted in News | Tagged , , | 2 Comments

Poster on “Computer Cluster Design Automation Using Web Services”

Posted on June 19, 2012 by Konstantin S. Solnushkin

I have just presented my new poster, “Computer Cluster Design Automation Using Web Services“, at the International Supercomputing Conference (ISC’12) in Hamburg.

The poster session moved this year to the exhibition hall. If you were there but missed it, use the link above for a PDF.

Posted in News | Comments Off on Poster on “Computer Cluster Design Automation Using Web Services”

Lustre Strives to Go Mainstream

Posted on June 13, 2012 by Konstantin S. Solnushkin

DDN EXAScaler. Image: DDN

Do you remember Chroma, the software from Whamcloud that allows to deploy and manage Lustre installations with a web interface?

In April 2012, DataDirect Networks (DDN) announced they have successfully integrated Chroma into their EXAScaler storage system. I believe it is a major step towards the wider recognition and adoption of Lustre, and not just in the HPC segment but, more importantly, in the general computing community as well.

One particular feature of EXAScaler is that it can be attached via InfiniBand interface — isn’t that great?! This way Lustre can easily enjoy high bandwidth provided by the InfiniBand fabric. I can’t wait when InfiniBand enters commodity data centres. And it will be even better if InfiniBand and Lustre come together. Those two powerful technologies can benefit the commodity server segment very well.

But wait — there is more. Rich Brueckner of insideHPC recently interviewed Percy Tzelnic, the Senior Vice President of EMC Corporation about the ongoing collaboration between EMC and Whamcloud. I find this fragment rather exciting:

insideHPC: How will this collaboration help with the goal of getting the Lustre client included in the upstream Linux kernel?

Percy Tzelnic: EMC is interested in making Lustre readily consumable by our commercial HPC customers. EMC is collaborating with the Linux community and Whamcloud to enable the availability of Lustre client software as part of the standard Linux distribution.

Just imagine — no more specific kernel version requirements or kernel patches, all your Lustre client functionality will be available on your ordinary server. That’s great news, indeed.

Posted in News | Tagged , , , , | Comments Off on Lustre Strives to Go Mainstream

Open Hardware Paves the Way to Commodity Water Cooled Servers

Posted on May 17, 2012 by Konstantin S. Solnushkin

Google is making its own custom servers, but they don’t disclose many important details. On the contrary, Facebook is trying to provide as much information as possible about their own custom servers — in the hope it will benefit the whole community.

What both big companies seem to be currently missing is the emergent shift to water cooled servers, the change that is already taking place in the HPC world. The open hardware movement will allow reuse of best engineering ideas, including field-proved implementations of water cooling.

I was inspired to write this post after I saw “The Story of Send” yesterday, hyperlinked from Google’s main page. What grabbed my attention was this video about custom Google server technology: “Google Data Center Efficiency Best Practices. Part 5 — Optimize Power Distribution”.

There, Tracy Van Dyk, power engineer at Google, explains how they power their servers: instead of having a big UPS somewhere in the machine room, Google installed a small battery to every server. See for yourself in the following screenshot from the video:

Continue reading →

Posted in News | Tagged , , , , | Comments Off on Open Hardware Paves the Way to Commodity Water Cooled Servers

The Giants That Slowed Down

Posted on May 5, 2012 by Konstantin S. Solnushkin

During the recent decades, the US was able to attract the best scientists from all over the world in every field, including computing, by providing them with competitive salaries and benefits. But now research budgets are being cut, and the former rate of progress cannot longer be maintained for the United States.

In 2013, the US will receive modest funding in their quest for exascale, while the European Commission, on the contrary, doubled their supercomputer spending to 1.2 billion Euro up to year 2020. (source)

The Japanese parliament also was investigating if spending for the “K” supercomputer had been excessive.

Insufficient funding means inability to recruit the best talents. Now, the “human capital” that, figuratively speaking, knows no borders can flow to other countries.

Continue reading →

Posted in News | Tagged , | Comments Off on The Giants That Slowed Down

Three Fat-tree Alternatives for the SuperMUC Cluster

Posted on April 22, 2012 by Konstantin S. Solnushkin

SuperMUC is a cluster supercomputer currently being deployed at Leibniz Supercomputing Centre (LRZ) in Germany. As of today, 8,200 cores of 110,000 have been put into operation — it is a so-called migration system for early user access. (UPD: Deployment was complete in July 2012). SuperMUC consists of almost 9500 compute nodes and has a peak performance of circa 3 petaflops.

The machine was designed by IBM. They chose to build a 4:1 blocking fat-tree network. Yours truly compared their solution with two other possible alternatives using the fat-tree design tool described on this site.

 

(Image copyright by LRZ, taken from this page)

Continue reading →

Posted in Miscellaneous | Tagged , , | Comments Off on Three Fat-tree Alternatives for the SuperMUC Cluster