Scali Extends Linux Cluster Management to Storage

by Timothy Prickett Morgan

Linux cluster management software maker Scali said this week it has updated its Scali Manage and MPI Connect software for managing Linux clusters so it can reach beyond the cluster and manage the storage associated with a Linux cluster. By adding support for storage, Scali is trying to make life easier for administrators of Linux clusters in the high performance computing market.

According to Micah Waldman, director of product management at Scali, the company is seeing an increasing number of Linux clusters using clustered file systems. Waldman says administrators of the clusters want one toolset to manage their clustered file systems.

The simple fact is that managing a cluster of hundreds of nodes is not, without the right tools, as easy to manage as a smaller cluster of larger SMP machines and their associated storage area network. While Linux clusters are cheaper from the perspective of hardware and software costs, they have higher setup costs than traditional parallel clusters of RISC/Unix servers, have more complexity, and people are less familiar with them. Attacking these problems is why Scali introduced the Scali Manage suite of tools for server nodes in the clusters; Scali also sells its Scali MPI Connect networking software that implements the MPI protocol used in parallel computing architectures and which gives a single point of management for the MPI network, just like Scali Manage gives a single point of installation, configuration, and management for the Linux nodes in an HPC cluster. Scali MPI Connect presents HPC applications with a single profile to reach out into nodes in the network, and is agnostic as far as the network transport that is actually used to carry MPI messages from node to node in the cluster--it can be InfiniBand, Ethernet, Myrinet, SCI, or even the internal SMP buses of big SMP servers. But the Scali Manage and Scali MPI Connect only solve two-thirds of the problem, says Waldman. The storage has to be managed, too.

Compute clusters can store their data sets directly on each node of the server, which is an inexpensive way to do it and is particularly good for compute-intensive workloads. But feeding 4 or 5 terabytes of information to 512 nodes in a Linux cluster is not an easy task. So cluster designers have opted instead to hook network-attached storage arrays to their clusters and allow nodes to share data sets. However, while this is good for some applications, it can be relatively expensive and has limited throughout. On a Gigabit Ethernet link between server nodes and the NAS, clusters are lucky to see a sustained throughput of 500 MB/sec out of the NAS and into the cluster. This is why clustered file systems--which essentially means taking a cluster of machines similar to a compute cluster and equipping it with a file system that spans the nodes and makes it look like a single NAS as far as the real compute cluster is concerned--have become more popular. With a clustered file system, says Scali, you can get a sustained data rate of about 150 MB/sec per node in the boxes that support the clustered file system, and the limit to bandwidth is really the limit of the bandwidth you use to link your compute nodes to the file system nodes. The clustered file system has reasonable costs, decent performance, and is scalable. If you need more throughput, you add more nodes to the file system cluster and more links into the compute cluster. (The typical ratio is one storage node for every 16 compute nodes, plus one node for housing storage metadata.)

Because clustered file systems are probably the way HPC clusters will use storage, Scali this week is introducing Scali Manage/Storage, an add-on module for the new Scali Manage 4.4 release that also comes out this week. Specifically, Scali will be supporting the configuration and management of the open source Lustre cluster file system in conjunction with Linux HPC clusters with this add-on. Scali has also partnered with Ibrix, the creator of the Segmented File System (SFS) for technical computing, so the Scali Manage/Storage add-on can configure and manage SFS as well. Scali will, in fact, sell and support the commercialized version of SFS, which is called the Ibrix Fusion Software Suite. There are other file systems Scali is looking at supporting, such as PolyServe's Matrix Server.

In addition to clustered file system support, Scali has also provided other tweaks for its software. The MPI Connect software has a new add-on called MPI Connect/HA, which will allow MPI interconnections to fail over to each other. For instance, if an InfiniBand mesh is the primary conduit for MPI messages between nodes, companies can now configure the MPI Connect software to use Gigabit Ethernet as a failover connection between nodes if an InfiniBand link dies. While this will degrade performance, it will allow jobs to finish. No one running a cluster wants to start a job over. And clusters are so big these days, that statistically, something is always failing in them somewhere. For example, in a cluster of 4,000 compute nodes with machines that have a two-year mean failure time, you will get five server failures a day. The updated MPI software also adds a layer of checksum above the TCP/IP socket layer to detect data errors and force a retransmission of data.

With Scali Manage 4.4, the company has also added better capacity planning tools and has preconfigured a set of best practices into the system for monitoring the components of the cluster and how they are performing. The software now also supports Red Hat Enterprise Linux 4, the latest Linux 2.6 version, as well as the existing RHEL 2 and 3; Novell's SUSE Linux Enterprise Server 9 has also been certified to run Scali Manage on Itanium-based nodes. Scali has also added some new hardware support, including the Silicon Graphics Altix 350, Hewlett-Packard's ProLiant BL20p G2, BL20p G3, and BL30p G3 Xeon-based blades and its new BL25p and BL35p Opteron-based blades. The HP two-way rx1620 Itanium server and the two-way DL385 Opteron-server have also been certified, as have Dell's PowerEdge SC1425, 1850, and 2850 Xeon servers and Sun Microsystems' Sun Fire V20z and V40z Opteron-based servers.