Distributed file systems for Raspberry and Banana Pi: LizardFS, GlusterFS and OpenAFS

1. LizardFS

• A fairly well hyped fork of MooseFS.
• Tried first, before it became available as a Debian package, therefore compiled from source.  It's not available in the Wheezy repositories for RPi but is available in later distributions.
• Compilation
• For compilation it needs asciidoc.  When asciidoc is installed it installs texlive.  Therefore, it takes up a huge amount of disk space.
• Compilation may need more swap space
• Surprisingly easy to set up, I used a PDF presentation which was available at the time.
• Choose a master server and add mfsmaster to each node's host file
• Make lizardfs_chunks directories for storage and add them to just that server's /etc/mfs/mfshdd.cfg.  Directories for removal can be marked by prefixing them with *.  Restart the chunkserver and it's done.
• Nice web interface, let's you keep an eye on all computers and their disk usages
• Number of replicas can be specified for each individual file, this is clever
• It worked OK to start with, including for making mencoder videos from security camera images
• Unfortunately, play back of high definition videos suffers from stutters.
• After some time, I got a lot of errors:


• I/O errors during rsync
• vim cannot save it's temporary files
• mencoder cannot create a movie file compiled from security camera stills
• I presume this is due to the Banana Pi being well short of the minimum requirements for a LizardFS master server 

2. GlusterFS

• Available in Debian, including Wheezy.  No need to compile it!  Except if connecting Wheezy to Jessie, because the differences in GlusterFS-sever versions lead to endless "peer probe: failed: Error through RPC layer, retry again later" errors.
• This system uses no metadata.
• Peers all act as a failover for each other, I think.
• Built-in NFS server - my RPi Model A needed this to connect with because somehow the actual GlusterFS client didn't work..
• Add peers with gluster peer probe HOSTNAME.
• Add storage 'bricks' with gluster volume add-brick Volume_name_eg_GlusterFS HOSTNAME:/directory/intended/for/storage/Volume_name_eg_GlusterFS.
• Check free space with gluster volume status GlusterFS detail
• Performance - to be tested!
• rsync - in progress, looks good so far, ran it a few times though.
• video - not yet tested
• mencoder - not yet tested 

Problems
Breaks when debian is upgraded on any of the servers
Interferes with NFS
Gave up.

3. OpenAFS
I didn't bother trying.

4. NFS
NFS is not a distributed file system.  But if you can split up your data (e.g. videos on one big hard drive, other stuff on another server) then you can avoid all the hassle of distributed file systems.  This is what I'm defaulting to now.