Shared directory over network of few Raspberry PIs, Odroid and a laptop

tl;dr

  • how to have a directory of files transparently synced between many servers
  • How to setup GlusterFS on 2 Raspberry PIs, Odroid and Linux PC
  • Had to use same version (3.6.x) everywhere; compiled from sources
  • have a backup as a side effect :-)

Intention

When working with multiple servers I had a need to have 1 directory available on each machine that could be shared among all servers. It should be automatically synchronized when server boots up. Each file modification/creation should be pushed to other servers too.

The storage itself should be redundant – backup for “free”. When all-but-one server dies: I still can access and modify files.

Decision

I will use distributed filesystem: GlusterFS. Meant for high volume traffic.
I will use it to share 10MB of ~/etc and ~/bin/ only ;-)

Setup

Raspberry PI 1 hostname: rpi-0, OS: Raspbian
Raspberry PI 2 hostname: rpi-2, OS: Raspbian
Odroid XU3 Lite, hostname: odroid, OS: Ubuntu 14
Laptop PC, OS: Debian Jessie

APT vs ./configure && make && make install

Unfortunately GlusterFS packages that come with Raspbian (R PI OS) and Ubuntu 14 (Odroid) and Debian Jessie are completely different in versions (3.2.x, 3.4.x). They’re all 3.x and install & run fine. However Odroid version wasn’t able to mount remote filesystem. I had to abandon using system packages in order not to waste any more time.

GlusterFS 3.7.x source code on Raspberry didn’t work with one dependency (liburcu), so I’ve used 3.6.x release with success.

Steps

Install the latest 3.6.x release

http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/

Install dependencies

sudo apt-get install libacl1-dev flex bison libreadline6-dev libxml2-dev libxml2 liburcu-dev liburcu1 python-dev make gcc flex bison libssl-dev liblvm2-dev libglib2.0-dev libxml2-dev python2.7 python2.7-dev autoconf libtool attr

Install

./configure --prefix=/site/apps/glusterfs-3.6/ && make && sudo make install
ln -s /site/apps/glusterfs-3.6/ /site/apps/glusterfs/

Compilation took:

  • 75 min on Raspberry PI 1
  • 26 min on Raspberry PI 2
  • 4 min on Odroid XU3
  • 2 min on Intel i7

Setup GlusterFS

mkdir -p /site/apps/glustervol #glusterfs server storage, not meant for direct access
mkdir /mnt/glusterfs/          #mount point of gluster filesystem, you put your files here
sudo export PATH=$PATH:/site/apps/glusterfs/bin/:/site/apps/glusterfs/sbin/                       #you might put it in /etc/profile or visudo (secure_path)
sudo update-rc.d glusterd defaults

sudo /etc/init.d/glusterd start #on every machine, or simply reboot

Setup pool of GlusterFS servers

In order to distribute data (files) accross multiple GlusterFS severs and support failover (in case some servers are down, files are still accessible):

sudo gluster peer probe rpi-2
sudo gluster peer probe odroid
sudo gluster peer probe rpi-0
sudo gluster peer info  #you should see all machines except host

 Create distributed volume

sudo gluster volume create glustervol replica 3 transport tcp rpi-0:/site/apps/glustervol/ rpi-2:/site/apps/glustervol/ odroid:/site/apps/glustervol/ force
sudo gluster volume start glustervol
sudo gluster volume info

Mount on remote machines

sudo mount.glusterfs rpi-0,rpi-2,odroid:/glustervol /mnt/glusterfs/

Mount on startup in /etc/fstab:

#GlusterFS
rpi-0,rpi-2,odroid:/glustervol /mnt/glusterfs/ glusterfs defaults,nofail,_netdev 0 0

The way it’s mounted (several hostnames provided) will prevail situations when some of nodes are down. Mounted files will be transparently remounted to other available node. Wow!