A client had a catastrophic fileserver failure resulting in the loss of a significant amount of important data. As a result, I was called in to setup an automated offsite backup. Due to many factors, I decided to implement a DroboFS instead of a hosted cloud or regular *nix fileserver.
Using the Drobo, there were a couple of gotchas. It doesn't have sshd enabled, no bash shell, no rsync (or rsnapshot which would make this easier) and more.
I will describe how I setup an automated, password-less, incremental differential, hard-linked backup using rsync across ssh.
Whats an incremental differential hard-linked backup?
It is essentially the same as Apple's Time Machine system. We create a backup using only the files and directories that have changed, but still store it as a full backup on the backup media. In other words, every time we backup, we will only transfer what has changed, but if you look at that backup on the remote computer, all the files will be there.
How do we do this? Utilizing hard-linking in the filesystem. Essentially we are creating two "pointers" that reference the same file (inode). That way each backup folder has its own copy of the file, but we only store the file once. If you run the command stat on a file or directory, you can see in the output how many links to that inode there are, or run ls -i to see the inode you are pointing to. New versions of that file won't overwrite the old ones, the new backups will reference a different inode, keeping your historical data.
This helps for several reasons. When we need to restore the entire filesystem, we can just move all the files from one directory, rather than merging the incrementals into a delta and applying it to the full backup to get our latest system and we only transfer what has changed which dramatically reduces bandwidth, time, and space.
Let's get started...
First things first, setup your Drobo as the instructions specify and be sure to enable DroboApps. Create a share on the Drobo for your backups. I'll name mine backup, if you use a different name, edit the below scripts as necessary.
You will then need to install the Dropbear SSH and Rsync apps for the following to work. [Note: You do not need the Apache or Admin apps, but you might find them useful.]
At this point you should be able to login via SSH to your Drobo using the default SSH credentials, at the time of writing for the DroboFS its root and root. We'll update the root SSH password using the following command:
/mnt/DroboFS/Shares/DroboApps/dropbear/root_passwd
This will reset and persist our root password between reboots.
[Note: I know that allowing root SSH access is not ideal but I have so far been unable to stop the drobo from allowing it if SSH is enabled... perhaps a commenter will have a solution?]
Now we need to edit the rsync config file:
vi /mnt/DroboFS/Shares/DroboApps/rsync/rsyncd.conf
You should see one share named [drobofs]. We want to replace it so that the file looks like:
uid = root
gid = root
pid file = /mnt/DroboFS/Shares/DroboApps/rsync/rsyncd.pid
[drobofs]
path = /mnt/DroboFS/Shares/backup
comment = Backup Share
read only = false
[Note: if you named your backup share different, you'll need to specify it here.]
Alright, now to provide password-less login. From the machine that has the data you want backed up, generate your SSH keys. You will likely not want to use a passphrase and I'll leave you to secure SSH as you see fit (perhaps read up about forced-commands-only).
user@server:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
f2:13:b7:23:75:da:4e:35:a8:32:61:af:43:e1:a0:53 user@server
Copy what is contained in id_rsa.pub. Now, on the Drobo we have to create the authorized_keys file to enable us to login with the key we just created. The required directory and files do not exist by default so, from the Drobo:
mkdir ~/.ssh
vi ~/.ssh/authorized_keys
Now copy the contents of id_rsa.pub into the authorized_keys file. You should now be able to SSH from the server to the Drobo without needing to enter a password.
Now that our apps are setup on the Drobo and communications will be seamless, we're going to create two scripts, one to rotate and manage the backups on the Drobo and one to do the actual rsync-ing.
The first script will reside on the Drobo in the root of the backup folder. In our example:
/mnt/DroboFS/Shares/backup
I'll call it rotate_backups.sh and it'll look like the following. Edit as necessary for your own purposes, and please note, the Drobo doesn't have the bash shell, so we're using good old #!/bin/sh
#!/bin/sh
# How many backups would you like to keep, each time you run
# the backup script, a new one will be created, so if you want:
# Daily for a week, script goes cron daily and enter 7.
# Hourly for 3 days, script goes cron hourly and enter 72 (24 hours x 3 days)
NUMOFBACKUPS=7
# Where are we backing up to?
BACKUPLOC=/mnt/DroboFS/Shares/backup
# Delete the oldest backup
NUMOFBACKUPS=`expr $NUMOFBACKUPS - 1`
if [ -d $BACKUPLOC/backup.$NUMOFBACKUPS ]; then
echo "delete backup.$NUMOFBACKUPS"
rm -Rf $BACKUPLOC/backup.$NUMOFBACKUPS
fi
# Move each snapshot
while [ $NUMOFBACKUPS -gt 0 ]
do
NUMOFBACKUPS=`expr $NUMOFBACKUPS - 1`
if [ -d $BACKUPLOC/backup.$NUMOFBACKUPS ] ; then
NEW=`expr $NUMOFBACKUPS + 1`
mv $BACKUPLOC/backup.$NUMOFBACKUPS $BACKUPLOC/backup.$NEW
echo "Move backup.$NUMOFBACKUPS to backup.$NEW"
fi
done
This script will delete your oldest backup and move the others down the line... so your latest backup will always be backup.0, 1 run old is backup.1, 2 runs old is backup.2, etc.
Now that our backups are managed properly we will create the actual rsync script. This will be run on the computer you want to back up from, and I suggest you add it to the crontab so it runs as often as you want it to. Edit BDIR, BSERVER, REMOVEDIR, and EXCLUDES as necessary.
#!/bin/sh
# Directory to backup.
BDIR=/data_to_backup
# Remote server (should match the password-less SSH credentials
# we setup earlier).
BSERVER=root@drobo
# Full path of the directory on the Drobo to backup to.
REMOTEDIR=/mnt/DroboFS/Shares/backup
# A list of files or directories to exclude.
EXCLUDES=/home/user/backup/excludes.txt
# Activate our rotate_backups script on the drobo.
ssh $BSERVER $REMOTEDIR/rotate_backups.sh
OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES --delete -av --rsync-path=/mnt/DroboFS/Shares/DroboApps/rsync/rsync"
# Do the rsync.
rsync $OPTS -e 'ssh -p 22' --link-dest=../backup.1 $BDIR $BSERVER:$REMOTEDIR/backup.0/
Add that script to your hourly/daily/whatever you want crontab and you should be good. The first time you run it you'll be transferring the entire backup directory so it might take a long time and you'll have an error message about --link-dest backup.1 not existing, but you can ignore that. Subsequent backups should run without a hitch.