HP, Red Hat Launch Sophisticated File Systems for Linux

by Timothy Prickett Morgan

While everyone seems to be thrilled by the prospect of clustering together relatively inexpensive Lintel iron for database or number-crunching work, managing a cluster is a bit of a nightmare and no one talks about that so much. Tuning network interconnections is a pain, and getting compute nodes to run at reasonable utilizations is also tough. Perhaps more than anything else, getting all of those nodes access to the data they need to compute is particularly difficult.

As is usually the case, the high performance computing centers of the world are leading the way when it comes to developing global or clustered file systems that work with Linux boxes to try to solve the data access and data management issues. While there are many such file systems in development (Silicon Graphic's XFS and IBM's JFS are but two that are popular with Linux), Hewlett-Packard and Red Hat have recently jumped into the game with their own offerings.

HP has not developed its own file system so much as make use of another open source file system as the heart of a sophisticated storage cluster for feeding Linux server clusters. Specifically, HP has chosen the open source Lustre file system, developed by Cluster File Systems of Medford, Massachusetts, as the basis of its Scalable File Share (SFS) product, a storage cluster (which HP calls a storage grid) that was announced a few weeks ago at the International Supercomputing Conference in Heidelberg, Germany. SFS is a collection of HP ProLiant servers running the Linux operating system and the Lustre file system that act as controllers for HP Enterprise Virtual Array (EVA) and Modular SAN Array (MSA) RAID disk arrays. What makes Lustre interesting is that this parallel file system automatically stripes data across arrays and disks, thereby improving performance and making data highly available; it also separates file meta data and storage allocation meta data (information about where files are located logically and where the data comprising those files are located physically as it is striped across drives and arrays in the storage cluster) from the data itself, allowing this information to be populated across the storage cluster. By doing this, the HP SFS storage cluster can support over 1,000 Linux compute nodes and hundreds of terabytes of data and still offer very high sustainable I/O bandwidth.

While HP didn't create Lustre, it has helped to make the open source file system better. Cluster File Systems was founded in 2001 by Peter Braam, who has been a senior faculty member in computer science at Oxford and Carnegie Mellon University and who has had senior technical positions at Red Hat, Turbolinux, and a number of other companies. In June 2002, HP was tapped by Sandia National Laboratory, Los Alamos National Laboratory, and Lawrence Livermore National Laboratories (three of the big government-sponsored HPC centers in the United States) to work with Cluster File Systems and Intel to make Lustre a more rugged product suitable for production. The 11 teraflops "MPP2" Linux-Itanium cluster at the Department of Energy's Pacific Northwest National Laboratory has a 53 TB SAN that runs Lustre. Each Linux node in the cluster can write data to the file system at a rate of 650 MB/sec, and the aggregate I/O bandwidth of the whole cluster has been measured at 3.2 GB/sec. HP claims that its implementation of Lustre in the SFS offers 100 times the bandwidth of the NFS file system commonly used in the Linux and Unix systems that comprise parallel HPC clusters.

While bandwidth is important in the MPP2 and similar Linux clusters, the real benefit of Lustre is that its architecture means that cluster administrators do not have to copy datasets to the 900 Linux compute nodes in the cluster; the file system also has mirroring and multipath capabilities, which means that if a disk or array fails, the Linux servers in the compute farm can keep doing their work. All the bandwidth in the world doesn't mean anything if the administration penalty for using a technology is too high. If this were not the case, clusters would have taken over commercial computing by now. But they did not, mainly because SMP servers have significantly lower administration costs than do clusters of similar power. At least for now. Developments like Lustre hope to change that equation.

The HP SFS solution is the productization of the work that HP and Cluster File Systems have been doing for all of these government labs. The SFS product has high-end EVA arrays or midrange MSA arrays, the latter of which has more modest virtualization capabilities. These virtualization features (snapshotting and remote copy) are not necessarily useful in a Lustre storage cluster, given how Lustre represents another kind of storage virtualization. These arrays are connected to ProLiant servers running Linux and the Lustre file system, which are in turn connected to Linux compute nodes through Ethernet, Gigabit Ethernet, Myrinet, InfiniBand, or Quadrics ELAN4 interconnections. HP delivers the whole shebang on as the SFS, which is a finished, integrated product. What it does not do is provide list prices for SFS.

Red Hat Goes Global

Over at Red Hat, the hot new thing is the Global File System that the commercial Linux distributor has begun selling this month. Red Hat acquired GFS when it bought Sistina Software, a Minneapolis, Minnesota storage expert, for $31 million in December 2003. While Sistina offered a commercial variant of the GFS software, the OpenGFS Project open source variant has made the software available in parallel and has done its own modifications to the GFS code. Red Hat has released its own open source version of the GFS product, which is tuned for its own Red Hat Enterprise Linux 3 variant, on its own site. Red Hat and OpenGFS project members are meeting on July 29 in Minneapolis to work out how the two open source development teams will work together from here on out, now that Red Hat has delivered its implementation of GFS.

GFS is aimed at allowing clusters of machines to share data, where that data is either inside the server nodes in the cluster or residing in a storage area network that all of the nodes link into. Red Hat says that its implementation of GFS is appropriate for running clustered databases such as Oracle 9i RAC and 10g for commercial transaction processing, high availability clustering for failover (in conjunction with Red Hat's Cluster Suite, which is based on the open source Piranha project), and high performance computing clusters based on Linux. Red Hat GFS costs $2,200 per server, and integrates with Red Hat's WS, ES, and AS variants of its Enterprise Linux 3. The WS support is important since many HPC Linux clusters use the workstation variant of Linux, not the server versions, since all they really want is the Linux kernel, the Fortran and C compilers, and some libraries. Red Hat GFS is supported on X86 (32-bit and 64-bit), Itanium (64-bit), and Opteron (32-bit and 64-bit) processors. It is unclear when or if it will be available for other architectures, such as IBM's Power-based i5 and p5 servers. It is also unclear how the open source version of Red Hat GFS will work with other Linux implementations. That's probably what Red Hat and the OpenGFS Project are going to talk about later this month.