Fat-tree networks work very well with InfiniBand hardware. At the same time, a fat-tree built with Ethernet switches may not always work: this is because switches must be able to discover multiple paths in a topology and balance traffic among them.
For topology discovery (detecting multiple paths), the switches must support OSPF protocol, or alternatively IS-IS or BGP protocols. For balancing traffic among those links, a standard called ECMP must also be supported.
If your Ethernet switches are outdated and support the spanning tree protocol (STP) rather than OSPF and friends, they cannot detect multiple redundant paths. If they don’t support the ECMP standard, they cannot utilise those paths. Only one path will be selected and used; the remaining ones will be disabled, bandwidth and resilience will be limited, and the fat-tree magic will just not happen.
Dell Networking Z9500 Ethernet switch with 132 40 Gigabit Ethernet QSFP+ ports. It supports all standards required to build fat-tree networks. Source: Dell.
This alphabet soup of protocols is not easy to navigate (and why would one want to navigate a soup?), so there is now a page that simply lists Ethernet hardware that is known to support fat-tree topologies out of the box: Fat-Trees with Ethernet Switches. You can also get there through the site menu.
Currently it lists switches made by Mellanox and Dell. If you have comments or updates, please leave them at that page.
