Distributed Filesystems Roundup

Revision $Id: distributed_filesystems.rhtml 187 2007-12-19 15:28:28Z jashmenn $

There are a wide variety of Linux distributed filesystems. With such a wide array of options it is often difficult to know where to start.

In our case, we were looking for a system with the following requirements

• Open-source or free
• Operates on consumer hardware over ethernet
• Could scale to minimum of 10TB
• Recovery if one of the nodes fail
• Ability to replicate for high-availability
• Ability to add new nodes on the fly and increase the storage pool easily
• Prefer ability to run in SuSE Linux
• Prefer to be able to run in a Virtual Machine using Xen

We’re just starting to learn about this kind of setup, so you may find a couple defects in this article. Please let me know any suggestions you have to keep this document up-to-date an accurate.

Here is a summary of a few of the options in light of the above criteria.

GlusterFS – The GNU Cluster FileSystem

License: GPL

Pros:

• Completely distributed. No centralized meta-data sever like Lustre
• Scales on Demand – what does that mean? can add new nodes? how?

Cons:

• Young
• Documentation falls apart when talking about healing a failed brick

Notes:

• Replication is a plugin (they call it translator)

Consumer Hardware: Yes

Max Space: ?

Recovery Mode: ?

Replication: plugin

Adding new nodes: ?

SuSE Linux: ?

Community Activity:

• Users Mailinglist: reasonable
• Developers Mailinglist: poor
• Developers IRC channel: very active

Links: Gluster Home Page

Drbd

DRBD is a block device which is designed to build high availability clusters. This is done by mirroring a whole block device via (a dedicated) network. You could see it as a network raid-1.

License:

Pros:

• Seems to have been around for a while
• Widely known
• Integrates easily with Linux HA
• DRBD on top of LVM is possible, but you have to be very careful about when and which LVM features you use, and how you do it
• SuSE package available

Cons:

• Limited to two nodes

Notes:

Consumer Hardware: Yes

Max Space: Limited to two nodes

Recovery Mode:

Replication: Block-Level

Adding new nodes: Limited to two nodes

SuSE Linux: Yes. Packages are there by default. “SuSE officially does include drbd and heartbeat in its standard distributions, as well as in its fully supported SuSE Linux Enterprise Sever (SLES) 8. The most recent ``unofficial’’ SuSE packages can be found in Lars Marowsky-Bree’s subtree: ftp.suse.com/pub/people/lmb/drbd and its mirrors.”1

Links: Linux HA’s Drbd page

Community Activity: High

• Easy to find links
• Example of using Drbd with MySQL

MogileFS

License: Open Source

Pros:

• No single point of failure
• File classes (e.g. replicate .jpg 2 times, .txt 4 times)
• Local filesystem agnostic
• Used by LiveJournal

Cons:

• Made to be used for web-applications.
• Weak documentation

Notes: Made for web-applications so it doesn’t quite do what we need. Ruby interface could be good for some developers

Consumer Hardware: Yes

Max Space: No published limit

Recovery Mode: ?

Replication: File-level

Adding new nodes: ?

SuSE Linux: ?

Links: http://www.danga.com/mogilefs/

Community Activity:

• Ruby interface, makes Rails integration easy.

Starfish Distributed Filesystem

License: Semi-open

Pros:

• Can be mounted like any other drive in Linux or Mac OSX

Cons:

• Free up to 1TB. Pay for more

Notes:

Consumer Hardware:

Max Space: 1TB Free. None listed for pay

Recovery Mode: Filenames and paths are preserved, you can just cp them off1.

Replication: Yes

Adding new nodes: At runtime

SuSE Linux: ?

Links: http://wiki.digitalbazaar.com/en/Starfish_Distributed_Filesystem

Community Activity: ?

Openfiler

Openfiler is a browser-based network storage software distribution. Openfiler provides file-based Network Attached Storage and block-based Storage Area Networking.

License: GPL

Pros:

• Easy to setup and install
• Share management

Cons:

• Rumors say the interface isn’t all it could be.

Notes:

Consumer Hardware: Yes

Max Space: ?

Recovery Mode: ?

Replication: ?

Adding new nodes: ?

SuSE Linux: It is its own operating system

Xen: ?

Links:

• Openfiler Homepage

Community Activity:

• Development seems active

FreeNAS

License: BSD

Pros:

• Small, whole distro, easy to install
• Well established

Cons:

• It’s only a NAS so there’s no natural replication or redundancy

Consumer Hardware: Yes

Max Space: One Machine

Recovery Mode: ?

Replication: Not supported

Adding new nodes: Not supported

SuSE Linux: No, it is its own distro

Links: http://www.freenas.org/

Community Activity:

• Active

Notes:

• Seem’s well established
• But better for one NAS appliance on your network. We’re looking for something that has no single point of failure.

OCFS2

License: GPL

Pros:

• SuSE packages

Cons:

Notes: Included in Linux 2.6.19 Kernel

Consumer Hardware: Yes

Max Space:

Recovery Mode:

Replication:

Adding new nodes:

SuSE Linux:

Links:

Community Activity:

Lustre

License: Opensource

Pros:

• Very scalable

Cons:

• Central Metadata server creates a new single point of failure
• Complex setup

Notes: Talk about all the super computers that use this

Consumer Hardware: Yes

Max Space: Many Petabytes

Recovery Mode:

Replication:

Adding new nodes:

SuSE Linux: Yes

Links:

Community Activity:

GFS

License: GPL

Pros:

• Backed by RedHat
• Widely used
• Add storage on thefly

Cons:

• Only runs on RedHat

Consumer Hardware:

Max Space: Over 100 nodes

Recovery Mode:

Replication:

Adding new nodes:

SuSE Linux: No

Links: RedHat GFS

Conclusion:

TODO