Sun Says File Systems Are An Important Differentiator

Published: July 18, 2007

by Timothy Prickett Morgan

Sun Microsystems was originally known as a workstation vendor, and it became best known as a maker of servers and their associated Solaris Unix operating system. But Sun has always been an innovator in file systems as well as operating systems, and three decades of innovation has not changed that. Sun still believes that file systems matter, and its Zettabyte File System (ZFS) and the expanding role it could play in the IT industry are proof of this belief.

ZFS is one of the bright spots in Sun's storage business, which was bolstered considerably but, to some, unexcitedly by its $4.8 billion acquisition of tape and disk array maker Storage Technology two years ago. ZFS is also one of the more interesting features of Sun's Solaris 10 operating system, and even though it came out later than expected, the functionality inside this file system is very sophisticated. And that is why a number of operating system makers and now a clustered file system maker for the high-performance computing segment are looking to make use of ZFS.

ZFS was available as open source code as part of the Solaris 10 rollout in 2005, even though it did not make it into Solaris 10 proper until June 2006 as part of the 06/06 Update to that Unix operating system. ZFS is, as the name suggests, a 128-bit file system, which gives it an enormous addressable capacity. Sun is looking ahead, which makes sense when you consider that the UFS file system often used with Solaris is 30 years old; and NFS, which Sun created in 1984 to allow workstation clients to access data on servers as if it were on their own machines, is pretty old, too--and still pretty useful and an industry standard. ZFS, which was originally billed as Dynamic File Service when Sun first started talking about it in June 2004, aims to have a lot of fault tolerance and error correction built into the guts of the file system, but also has a lot of commands that makes it easier to manage storage volumes and disk pools, which have been virtualized and can co-exist with QFS, NFS, UFS, and other file managers and file systems.

"Our fundamental design is to do for disks what virtual memory did for main memory," explains Bill Moore, one of the co-creators of ZFS and a senior staff engineer at Sun.

In April, in keeping with its goal of taking all of its key software products open source, a number of storage-related bits of code related to ZFS were tucked underneath the OpenSolaris project, which is where the next rev of Solaris, code-named "Nevada," is being created. Specifically, the RAID Z algorithm created by Sun, which keeps two sets of parity data compared to RAID 5's one, was open sourced, as was the code that manages the creation of hot spare disks in a ZFS disk pool. Point In Time Copy and Remote Mirroring features for ZFS, NFS v4.1 (which is sometimes called parallel NFS), YANFS (formerly known as WebNFS, a piece of software that is written in Java and that implements the client half of the NFS v2, NFS v3, RPC, and XDR protocols) were also contributed to the open source community, as were drivers for iSCSI, OSD, and Fibre Channel peripherals (QLogic provided the open source FC drivers).

Given the features inside ZFS, it comes as no surprise that Sun is excited to see other companies and open source projects pick it up. According to Jeff Bonwick, who is a co-creator of ZFS with Moore and the storage chief technology officer at Sun, the open source FreeBSD variant of BSD Unix already supports ZFS in its distro and the NetBSD is adding ZFS support through Google's Summer of Code effort, whereby Google pays young nerds to do coding for a summer job. The future "Leopard" version of MacOS X Server also has ZFS support inside of its beta versions, but it remains to be seen if the file system will make it into the final release of Apple's server operating system. (MacOS is based on the BSD kernel and operating system, with a Mac user interface and other features plus a PowerPC compatibility emulation environment called Rosetta strapped onto it.) Google also ponied up the dough through its Summer of Code program last year to get ZFS ported to the FUSE framework for the Linux operating system.

Sun and Cluster File Systems, the company behind the open source Lustre clustered file system, have also announced that Lustre will now use ZFS underneath to provide the data integrity for Lustre. Lustre runs on Linux servers, and creates a clustered file system out of multiple server nodes with integrated disk arrays. Lustre scales up to petabytes of storage and, more importantly, can deliver the kind of high bandwidth that supercomputers (usually built for clusters of servers as well) require to run in a balanced fashion. By using ZFS, Cluster File Systems says that it can expand Lustre to scale up to trillions of files and exabytes of storage capacity. It is not clear if Lustre is being ported to Solaris, but it will certainly remain available on Linux, which is the dominant operating system in the HPC market today.

Sun's interest in having ZFS used on other platforms goes beyond just having lots of ports. Sun is also shooting for broader file compatibility. "We want ZFS to be ported to other platforms, but we also want the data stored in ZFS to be available and readable by other platforms," explains Bonwick. Of course, that would mean getting AIX, HP-UX, and Windows to support ZFS as well. Sun would not say if ZFS is part of the co-development and interoperability work between Sun and Microsoft after they buried the hatchet two years ago. But Bonwick did offer an observation of the possibilities. "If ZFS goes on Windows, it becomes a ubiquitous standard as well as an interchange medium, much as FAT32 has been in the past and today."

Sun is now working on a number of features for ZFS that it believes will make it a differentiating feature for Solaris as well as any other platform that supports it. In late 2007 or early 2008, ZFS will get native data encryption support. ZFS already has a number of different native file compression algorithms, and these features are very popular. Native encryption certainly will be, too. Sun is also working on another feature called delegated administration. Usually, you have to have root access to a Unix or Linux server to administer file systems and their related storage. But with delegated administration, certain administration functions for ZFS can be relegated to specific Solaris containers (sometimes called zones), and the administrator responsible for those containers can change aspects of ZFS without getting root access, such as turning on encryption or compression, giving a file system more storage capacity, or setting storage quotas for users or applications. The delegated administration features can be accessed through NFS, too, which means the admin doesn't even need to be physically on the server to tweak stuff in ZFS.

RELATED STORIES

OpenSolaris Gets Lots of Storage-Related Code from Sun

Solaris 10 6/06 Update Ships with PostgreSQL, ZFS

Sun Previews Next Rev of Solaris 10

Linux Runtime, ZFS File System Still Coming for Solaris 10

Solaris 10 to Include Massive Dynamic File System

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot