Fat-tree and Torus Articles Now Available at arXiv.org

In case you wanted something more than an informal introduction into the world of fat-tree and torus networks — here you are. A formal problem statement (and solution!) in an academic form is given in these two articles hosted at arXiv:

K.S. Solnushkin, “Automated Design of Two-Layer Fat-Tree Networks”
(link: arXiv:1301.6179) (cite with BibTeX)

K.S. Solnushkin, “Automated Design of Torus Networks”
(link: arXiv:1301.6180) (cite with BibTeX)

arXiv is the leading open access repository of scientific papers.

Posted in News | Tagged , , | 2 Comments

Cluster Design Tools Updated (ver. 0.8.1a)

A new version of Cluster Design Tools was made available for download today. The change concerns the UPS sizing algorithm, making it yet more optimal.





Posted in News | Comments Off on Cluster Design Tools Updated (ver. 0.8.1a)

Real Cost Comparison of Fat-tree and Torus Networks

Thanks to Mellanox Technologies, our tool that designs fat-tree and torus networks now operates with real life prices for InfiniBand hardware. Mellanox kindly provided list prices for the previous generation of switches, InfiniBand QDR: these figures are not likely to change. The three switches are: Grid Director™ 4036 (36 ports), IS5100 Chassis Switch (18..108 ports) and IS5200 Chassis Switch (18..216 ports).

Remember the two things: (a) prices are for QDR InfiniBand hardware; for the most recent prices and the current InfiniBand FDR hardware, please contact Mellanox; (b) you can always download the tool and supply it with your own prices.

The main advantage of our fat-tree design tool is that it tries all possible configurations of modular (chassis) switches, including those where only some line cards are installed, hence recommending the most cost-efficient designs. Additionally, you can specify to the tool if your network must be expandable in the future, and up to how many hosts.

I decided to use the newly available prices to objectively compare costs of the following networks: non-blocking and 2:1 blocking fat-trees and torus networks.


Cost comparison of fat-tree and torus networks with up to 3,888 nodes

Continue reading

Posted in News | Tagged , , , , | 3 Comments

Latency Everywhere

People from the high-performance computing field have a clear understanding that performance of technical systems (of various nature) is characterized by two metrics: throughput and latency. People in other fields sometimes focus on either throughput or latency alone.

MV Colombo Express, one of the largest container ships in the world. Image source: Wikipedia

For many years it has been a well-accepted truth in the HPC field that latency in the computer network does matter. But with the proliferation of Gigabit Ethernet on desktops, people started to expect their network connection to function at a high bandwidth. It quickly turned out that big network latencies can easily limit available bandwidth. That’s how ordinary people learned about bandwidth, latency, and their interdependence. Continue reading

Posted in Miscellaneous | Tagged , | Comments Off on Latency Everywhere

“I’ll Start My Own Supercomputer Conference”

Remember that moment from Futurama when Robot Bender promises to set up his own theme park? Forget about the theme park, because is seems that we now have several supercomputer conferences. Why? And does it do us any good?

I am a member of the ACM, and in their recent mailing I received news about upcoming events in the field of high-performance computing. What caught my attention was the not-so-major event that, nevertheless, proudly called itself the “International Conference on Supercomputing”. Continue reading

Posted in Miscellaneous | Tagged | Comments Off on “I’ll Start My Own Supercomputer Conference”

Finally, A Topology-Aware MPI Implementation

Good news from the Supercomputing-2012 (SC12) conference: ten collaborators (including a talented team led by Dr. Dhabaleswar K. Panda) presented a paper on a new approach for assigning processes to compute nodes in InfiniBand networks.

Roughly, it works as follows: a plugin for OpenSM subnet manager retrieves a network topology by querying switches, topology information is passed to the MPI library (MVAPICH2), and finally the MPI library ensures that MPI processes that need to exchange large volumes of data are placed onto physically close compute nodes — i.e., within the minimum number of switch hops — or, ideally, within a single compute node.

The method allows to reduce the execution time of parallel applications by 6% to 15%, depending on the application and the number of MPI processes in a job: large-scale jobs appear to benefit more from the topology-aware placement.

As the new topology discovery scheme is capable of converting both fat-tree and torus topologies into tree-based representations, this technique should work for multi-dimensional torus networks as well, including 3D and 5D tori in IBM BlueGene. Initial results from the presentation seem to indicate this. As BlueGenes feature a proprietary network, and don’t have an InfiniBand-compatible subnet manager, another mechanism will be required to retrieve topology information here. Further exploration of torus based networks is the area of future work for the team.

Read the full paper here:

“Design of a Scalable InfiniBand Topology Service to Enable Network-Topology-Aware Placement of Processes”, by Hari Subramoni, Sreeram Potluri, Krishna Kandalla, Bill Barth, Jerome Vienne, Jeff Keasler, Karen Tomko, Karl Schulz, Adam Moody, Dhabaleswar Panda (PDF).

Posted in Reviews | Tagged , , , | Comments Off on Finally, A Topology-Aware MPI Implementation

How To Size Your UPS System: More Power, Igor!

We now have a tool to size UPS systems! Find it in the menu above, or use this link.

Battery shelves for the biggest (65 MW) UPS system in the world: Battery Electric Storage System (BESS) in Fairbanks, Alaska. Image source: GVEA.

(As for the headline, Igor is a character of Tom Holt’s novel, “Igor”)

Posted in News | Tagged , | Comments Off on How To Size Your UPS System: More Power, Igor!

System-on-Wafer: Integrated Circuit Packaging Considered Harmful

Silicon wafers contain enormous numbers of microprocessors. For example, a typical 300 mm wafer, as used by Intel, can contain about 130 dies of Intel Core i7 “Sandy Bridge-E” CPUs (use the formula, Luke, and substitute S=435 mm² for a die size).

Silicon wafers can contain lots of chips! Image by: Wikipedia user “Stahlkocher”. Source: Wikimedia Commons.

However, after packaging, CPUs take slightly more physical space. And then, density is dramatically lost again when you put CPUs onto a motherboard, and finally enclose that into a computer case. What used to be a thin silicon device measuring just 21×21 mm, now received a huge outer “package”. Of course, a motherboard conveniently integrates other important components such as memory modules and an interconnection network adaptor. But could we manage without integrated circuit packaging in the first place?

Seems that semiconductor engineers from Moscow, Russia have some bright ideas for us. Continue reading

Posted in Reviews | Tagged , , , , , | Comments Off on System-on-Wafer: Integrated Circuit Packaging Considered Harmful

Torus Network: As Round As It Gets

Fat-trees have great performance characteristics, but they can be costly for some installations. Blocking fat-trees can be used as an alternative, but if your computational task has locality of communications and is well suited to torus networks — why not to try them?

Our web service that was used to design fat-trees is now updated, and can now help you with torus networks, too. Try it here.

3D torus rendered by Fujitsu. (c) Fujitsu and RIKEN, 2009. Source: press release.

Posted in News | Tagged , , | Comments Off on Torus Network: As Round As It Gets

Fruits of Computing: Redefining “Green” in HPC Energy Usage

Suppose you are visiting a supercomputing centre for several days. Each day you submit your compute jobs and then go to a nearby tropical garden. While your jobs are being computed, you enjoy fresh tropical fruit, listen to exotic birds and read a recent issue of your favourite scientific journal.

Layard’s Parakeet, photo by Hafiz Issadeen. Source: Wikimedia Commons

Sounds too good to be true? But this could be our near future. In fact, modern supercomputers consume so much energy (and dissipate so much heat), that we must reuse that heat if we want to be green and be kind to Mother Nature. Continue reading

Posted in Ideas | Tagged , , , | 1 Comment