Shared directory over network of few Raspberry PIs, Odroid and a laptop

tl;dr

  • how to have a directory of files transparently synced between many servers
  • How to setup GlusterFS on 2 Raspberry PIs, Odroid and Linux PC
  • Had to use same version (3.6.x) everywhere; compiled from sources
  • have a backup as a side effect :-)

Intention

When working with multiple servers I had a need to have 1 directory available on each machine that could be shared among all servers. It should be automatically synchronized when server boots up. Each file modification/creation should be pushed to other servers too.

The storage itself should be redundant – backup for “free”. When all-but-one server dies: I still can access and modify files.

Decision

I will use distributed filesystem: GlusterFS. Meant for high volume traffic.
I will use it to share 10MB of ~/etc and ~/bin/ only ;-)

Setup

Raspberry PI 1 hostname: rpi-0, OS: Raspbian
Raspberry PI 2 hostname: rpi-2, OS: Raspbian
Odroid XU3 Lite, hostname: odroid, OS: Ubuntu 14
Laptop PC, OS: Debian Jessie

APT vs ./configure && make && make install

Unfortunately GlusterFS packages that come with Raspbian (R PI OS) and Ubuntu 14 (Odroid) and Debian Jessie are completely different in versions (3.2.x, 3.4.x). They’re all 3.x and install & run fine. However Odroid version wasn’t able to mount remote filesystem. I had to abandon using system packages in order not to waste any more time.

GlusterFS 3.7.x source code on Raspberry didn’t work with one dependency (liburcu), so I’ve used 3.6.x release with success.

Steps

Install the latest 3.6.x release

http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/

Install dependencies

sudo apt-get install libacl1-dev flex bison libreadline6-dev libxml2-dev libxml2 liburcu-dev liburcu1 python-dev make gcc flex bison libssl-dev liblvm2-dev libglib2.0-dev libxml2-dev python2.7 python2.7-dev autoconf libtool attr

Install

./configure --prefix=/site/apps/glusterfs-3.6/ && make && sudo make install
ln -s /site/apps/glusterfs-3.6/ /site/apps/glusterfs/

Compilation took:

  • 75 min on Raspberry PI 1
  • 26 min on Raspberry PI 2
  • 4 min on Odroid XU3
  • 2 min on Intel i7

Setup GlusterFS

mkdir -p /site/apps/glustervol #glusterfs server storage, not meant for direct access
mkdir /mnt/glusterfs/          #mount point of gluster filesystem, you put your files here
sudo export PATH=$PATH:/site/apps/glusterfs/bin/:/site/apps/glusterfs/sbin/                       #you might put it in /etc/profile or visudo (secure_path)
sudo update-rc.d glusterd defaults

sudo /etc/init.d/glusterd start #on every machine, or simply reboot

Setup pool of GlusterFS servers

In order to distribute data (files) accross multiple GlusterFS severs and support failover (in case some servers are down, files are still accessible):

sudo gluster peer probe rpi-2
sudo gluster peer probe odroid
sudo gluster peer probe rpi-0
sudo gluster peer info  #you should see all machines except host

 Create distributed volume

sudo gluster volume create glustervol replica 3 transport tcp rpi-0:/site/apps/glustervol/ rpi-2:/site/apps/glustervol/ odroid:/site/apps/glustervol/ force
sudo gluster volume start glustervol
sudo gluster volume info

Mount on remote machines

sudo mount.glusterfs rpi-0,rpi-2,odroid:/glustervol /mnt/glusterfs/

Mount on startup in /etc/fstab:

#GlusterFS
rpi-0,rpi-2,odroid:/glustervol /mnt/glusterfs/ glusterfs defaults,nofail,_netdev 0 0

The way it’s mounted (several hostnames provided) will prevail situations when some of nodes are down. Mounted files will be transparently remounted to other available node. Wow!

Raspberry PI arrived – setup howto

TL;DR

This is mainly a reference for myself how and what to install/configure on a fresh Raspberry PI

Hope it helps you too

Instructions

Have a Raspberry PI

  • on a 8GB card download and install an image: Raspbian
  • Power on Raspberry PI
  • First time installation
    • via GUI: maximize hard drive
    • set GPU RAM to 16
    • enable SSH
    • setup static DHCP address on home router
    • setup another system user (thanks to this post) and make sure it can do `sudo` without password prompt
      • adduser foo
        echo "foo ALL=(ALL) NOPASSWD: ALL" >>/etc/sudoers
        userdel pi
    • add yourself to disk group
      • sudo vigr # find disk, add your username
    • setup SSH
      • sshkey-gen
        remote$ ssh-add foo@..
        remote$ ssh-copy-id foo@..
        vim /etc/ssh/sshd_config
        PermitRootLogin no
        #http://blog.self.li/post/63281257339/raspberry-pi-part-1-basic-setup-without-cables
    • setup /etc/hosts
      • sudo vi /etc/hosts #add your fav hostnames here
    • Setup timezone
      • sudo tzselect
    • Update OS (2nd step might take a while)
      • sudo apt-get update
        sudo apt-get upgrade
  • Install useful software
    • sudo apt-get install vim samba screen mc git-core htop tig pydf nmap gqview iotop iperf nmap pydf tig cryptsetup ftp pv gitk evince curl w3m lynx xdu baobab mutt git fbset libpcre3-dev libboost-dev omxplayer libgstreamer1.0-0-dbg gstreamer1.0-tools libgstreamer-plugins-base1.0-0 gstreamer1.0-plugins-good gstreamer1.0-plugins-bad-dbg gstreamer1.0-omx gstreamer1.0-alsa dcfldd tcpdump zip subversion libudev-dev make build-essential cec-utils
    • sudo apt-get install
    • vim /etc/apt/sources.list
      deb-src http://archive.raspbian.org/raspbian wheezy main contrib non-free rpi
      sudo apt-get update
      mkdir ~/unrar-nonfree && cd ~/unrar-nonfree
      sudo apt-get build-dep unrar-nonfree
      sudo apt-get source -b unrar-nonfree
      sudo dpkg -i unrar_4.1.4-1_armhf.deb
      cd && sudo rm -rf ~/unrar-nonfree
      sudo apt-get install unrar-free
  • Setup of my dotfiles (vimrc, bashrc for better console cmd history)
    • cd; mkdir -p github; git clone https://github.com/nthx/dotfiles.git; cd dotfiles; ./install
  • setup WiFi
    • Router: Disable “Enable AP Isolation” – this prevented my laptop accessing RPi network
    • Bus 001 Device 005: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter
    • cp /etc/network/interfaces /etc/network/interfaces.backup
      cp /etc/wpa_supplicant/wpa_supplicant.conf /etc/wpa_supplicant/wpa_supplicant.conf.backup
    • #vim /etc/network/interfaces
      auto lo
      iface lo inet loopback
      iface eth0 inet dhcp
      auto wlan0
      allow-hotplug wlan0
      iface wlan0 inet dhcp
         post-up iwconfig wlan0 power off
      wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
    • #vim /etc/wpa_supplicant/wpa_supplicant.conf
      ctrl_interface=/var/run/wpa_supplicant
      ctrl_interface_group=0
      update_config=1
      network={
       ssid="***-24"
       psk="***"
       id_str="rpi-0"
      }
    • sudo ifdown wlan0
      sudo ifup wlan0
      sudo wpa_cli status
    • In order to make Raspberry Pi automatically reconnect to Wifi after router down, use this solution:
      https://raspberrypi.stackexchange.com/a/5341/28218
      Works perfectly for me
    • For TP-LINK TL-WN725N card use this tutorial: https://www.raspberrypi.org/forums/viewtopic.php?p=462982#p462982
      • wget https://dl.dropboxusercontent.com/u/80256631/8188eu-v7-20150212.tar.gz
        wget https://dl.dropboxusercontent.com/u/80256631/8188eu-v7-20150818.tar.gz
        
        
  • Samba
    • cp /etc/samba/smb.conf /etc/samba/smb.conf.backup
      socket options = TCP_NODELAY
      workgroup = FOOBAR
      [pictures]
         comment = opt
         writable = no
         locking = no
         path = /mnt/foo/bar
         public = yes
    • service samba restart
  • Ruby
    • rvm (this is long, like hours of compilation)
      • https://rvm.io/rvm/install
  • SqueezeBox server
    • http://allthingspi.webspace.virginmedia.com/lms.php
  • DLNA Server
    • http://bbrks.me/rpi-minidlna-media-server/
    • apt-get install minidlna
      vi /etc/minidlna.conf
      #inotify=yes
      #db_dir=.... #setup to your hdd, not sd
      #log_dir=... #setup to your hdd, not sd
      #listening_ip=0.0.0.0 #though, my version failed on this - I had to disable and start /usr/bin/minidlnad -f /etc/minidlna.conf manually once
      #friendly_name=foo...
      #serial=bar...
      #notify_interval=895
      echo 120000 > /proc/sys/fs/inotify/max_user_watches
      sudo sysctl fs.inotify.max_user_watches=66538
      sudo service minidlna restart
  • PS3 Eye camera video straming
    • sudo apt-get install motion ffmpeg
      sudo vi /etc/default/motion # start_motion_daemon=yes
  • Console based bandwith speed test
    • wget https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
      chmod a+rx speedtest_cli.py ; ./speedtest_cli.py
  • Udev rules for auto-mounting HDD
    • ls -l /dev/disk/by-uuid/ #find your uuid and:
      sudo vi /etc/fstab
      UUID=41b81019-8838-4d22-9bf2-3673ca855ce7 /mnt/foobar ext4 noauto,user,sync,noatime,errors=remount-ro 0 2
    • udevadm info --name=/dev/sda1 --attribute-walk
      
      sudo vi /etc/udev/rules.d/81-my-drives.rules
      ACTION=="add", KERNELS=="sd*1", ATTRS{size}=="312581745", ATTRS{partition}=="1", SYMLINK+="foobar", RUN+="/bin/mount /mnt/foobar/"
      ACTION=="remove", KERNELS=="sd*1", ATTRS{size}=="312581745", ATTRS{partition}=="1", SYMLINK+="foobar", RUN+="/bin/umount -lf /mnt/foobar/"
      sudo udevadm control --reload-rules
      sudo udevadm test -a -p $(udevadm info -q path -n /dev/sda1)
  • Local DNS server for access of local servers by name
    • http://www.ducky-pond.com/posts/2013/Oct/how-to-setup-a-dns-server-with-powerdns-on-raspberry-pi/
    • sudo apt-get install pdns-serve
      sudo sed -i 's/# recursor=/recursor=8.8.8.8/g' /etc/powerdns/pdns.conf
      sudo sed -i 's/allow-recursion=127.0.0.1/allow-recursion=127.0.0.1,192.168.1.0\/24/g' /etc/powerdns/pdns.conf
      sudo service pdns restart
      sudo apt-get install dnsutils
    • vim /etc/powerdns/bind/example.com.zone
      $ORIGIN example.com     ; base for unqualified names
      $TTL 1h                 ; default time-to-live
      @                       IN      SOA ns.example.com hostmaster.example.com (
                                      1; serial
                                      1d; refresh
                                      2h; retry
                                      4w; expire
                                      1h; minimum time-to-live
                              )
                              IN      NS      ns
                              IN      A       192.168.1.15
      ns                      IN      A       192.168.1.15
    • nslookup example.com localhost
    • Then in your device in network configuration add Rpi IP address to list of DNS servers. Also in search domain enter: “example.com”
  • Firewall
    • sudo apt-get install ufw
      sudo ufw allow 22 #ssh
      sudo ufw allow 21 #ftp
      sudo ufw allow 53 #dns
      sudo ufw allow 137,138,139,445/tcp #samba
      sudo ufw allow 80 #www
      sudo ufw allow 8080 #www
      sudo ufw allow 111,2049/tcp #port mapper (nfs, glusterfs)
      sudo ufw allow 8554 #vlc
      sudo ufw allow 631 #cups
      #glusterfs
      sudo ufw allow 24007,24008/tcp
      sudo ufw allow 24009,24010,24011/tcp
      sudo ufw allow 49152,49153,49154/tcp
      sudo ufw allow 38465,38466,38467/tcp
      sudo ufw allow 24008/tcp
      sudo ufw enable
    • fail2ban – this I don’t yet cover here, as not yet sure how it works. Was banning myself too much ;-)
  • Software watchdog of network connectivity to your home router
    • */5 * * * * /home/foo/bin/soft-watchdog-wifi.sh
    • In order for Wifi to reconnect see this solution: http://raspberrypi.stackexchange.com/questions/4120/how-to-automatically-reconnect-wifi?newreg=e44a9ccb623b402094502f27c554e1c5
    • cd /etc/ifplugd/action.d/
      mv ifupdown ifupdown.original
      cp /etc/wpa_supplicant/ifupdown.sh ./ifupdown
      sudo reboot
  • Usenet with Sabnzbd
    • http://www.howtogeek.com/142249/how-to-turn-a-raspberry-pi-into-an-always-on-usenet-machine/
      sudo apt-get install sabnzbdplus par2
  • FTP server for backup of my iPhone pics
    sudo apt-get install vsftpd
    sudo adduser ftpsecure
    sudo vi /etc/vsftpd.conf
    #listen=YES
    #allow_writeable_chroot=YES
    #local_umask=022
    #write_enable=YES
    #chroot_local_user=YES
    #nopriv_user=ftpsecure
    #chroot_list_enable=YES
    #chroot_list_file=/etc/vsftpd.chroot_list
    
    sudo vi /etc/vsftpd.chroot_list
    foo
    sudo service vsftpd restart
  • For iPhone iOS app PhotoSync in order to backup pics/videos via ftp this has to be also done (if one wants other than ~/Pictures dir only)
    mount --bind /foo/bar/archived-synced/ /home/foo/Pictures/
  • Photos management

    https://www.linux.com/learn/how-sort-and-remove-duplicate-photos-linux

    sudo apt-get install  fdupes exiftool
  • NFS
    sudo apt-get install nfs-kernel-server nfs-common portmap
    sudo update-rc.d rpcbind enable && sudo update-rc.d nfs-common enable
    sudo vi /etc/exports
    /mnt/foo/bar/ *(rw,sync,no_root_squash,no_subtree_check)
    sudo exportfs -av
    sudo service nfs-kernel-server restart
    sudo service rpcbind restart
  • HDD maintenance
    sudo apt-get install smartmontools gsmartcontrol
    sudo smartctl --all /dev/sdd
    
    touch /forcefsck
    sudo shutdown -rF now
    
    sudo apt-get install gnome-disk-utility
    sudo gnome-disks #somehow on raspbian it's missing :-?
    sudo gsmartcontrol
  • crontab logs
    • sudo vi /etc/rsyslog.conf
      #ucomment #cron
      sudo /etc/init.d/rsyslog restart
  • security
    • list all services running on the system
      sudo service --status-all
  • screenfetch – show system details on a console
    wget https://raw.github.com/KittyKatt/screenFetch/master/screenfetch-dev
    chmod a+rx screenfetch-dev
    ./screenfetch-dev
  • dd with progress
    dd if=/dev/zero | pv --size 16g | dd of=/dev/sdb
    dcfldd if=/tmp/in of=/tmp/out
  • camera
    sudo modprobe bcm2835-v4l2
    echo bcm2835-v4l2 >> /etc/modules
    sudo apt-get install vlc v4l-utils motion
    raspivid -o - -t 0 -n -w 1280 -h 720 | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8888/}' :demux=h264
  • sound
    echo snd-bcm2835 >> /etc/modules
    vi /etc/asound.conf
    
    pcm.!default {
      type hw    
      card 0 }
    ctl.!default {
    type hw
    card 0   }
  • print server
    sudo apt-get install cups
    sudo cupsctl --remote-any
    sudo usermod -a -G lpadmin nthx
    sudo apt-get install avahi-discover #for air print
    #canon ip4700 works
    sudo /etc/init.d/cups restart 
    browser http://localhost:631/
    #add your driver
  • purge some extra space
    sudo apt-get install localepurge deborphan wajig 
    sudo localepurge
    sudo apt-get autoremove
    sudo apt-get autoclean
    sudo apt-get clean
    sudo apt-get purge wolfram-engine minecraft-pi sonic-pi gnome-accessibility-themes gnome-themes* gnome-icon-theme-extras gnome-icon-theme-symbolic gnome-games*
    
    wajig large #and analyze..
    baobab #and analyze
  • Process Canon RAW files
    sudo apt-get install ufraw ufraw-batch
    ufraw-batch --out-type png *.crw

 

  • Power off Raspberry PI
    sudo poweroff

 

Upgrading to Raspberry PI 2

  • on existing Rpi1 apt-get update && apt-get upgrade && apt-get dist-upgrade
  • copy entire SD card to another dd if=/dev/mmcblk0 of=backup-of-mmcblk0
  • mount /dev/mmcblk0p2 /mnt/memcard
  • vi /etc/hostname #modify hostname
  • vi /etc/hosts #modify hostname
  • ssh-keygen
  • raspi-config