send feedback to webmaster @ schapiro . orglast modified 2008-05-29 12:21:58
| The Schapiro Center | Personal Schlomo Nadia Boris | Other Sites RKM Links |
| Small Projects | easyVCB | OpenVPN Gateway Builder | Rsync Backup Made Easy |
VMwareTools modules |
Version 1.4 improves the output (regarding run-time information) and fixes a bug where the definition of the current day changed during rbme execution. Lockfiles are now used to prevent parallel execution of the same RBME configuration.
Version 1.3 fixes an unfortunate bug that I introduced in 1.2. The bug would prevent successful backup when old backups had to be deleted.
Version 1.2 comes with many usability improvements:
Using rsync for backups is becoming more and more popular on UNIX/Linux systems. Especially the variant with hardlinks on the backup media to conserve disk space has proven itself to be a very reliable backup solution. The method is already widely published on the Internet (e.g. this is a nice introduction), so that I will save myself the trouble of repeating it.
In a nutshell you get perpetual incremental backups that appear as full backups (for each day) and thus allow easy restore or further copying to tape etc. After the initial full backup RBME will forever do incremental backups against the last backup to conserve time and network bandwidth.
RBME is a simple tool to assist in implementing a general backup solution based on rsync and hardlinks with an automated management of the disk space on the backup media. This is achieved by removing old backups from the backup media in a way which will ensure that (over the long run) the amount of backups should even out for all hosts involved.
This is also the major reason for me to write this script, I didn't find a suitable script (that would manage the backup space and also watch the inodes) when I needed it. The other reason is of course the fact that studying another script would take at least the same time as writing my own and in such cases I tend to write my own.
RBME expects you to provide a (large) backup space, preferably on a filesystem of it's own (it makes the disk space management easier). After configuring RBME with your backup path and the thresholds that seem appropriately for your environment, simply run rbme for each backup host. By default it will use rsync over ssh to copy that host to the backup filesystem (Use ssh -c blowfish-cbc to somewhat speed up your backups by choosing a faster cipher, if your security policy allows for that). Example:
# rbme host3 host2 host1
This example will simply backup the hosts host3, host2 and host1 to your backup space. Every day on which you run RBME will produce a new subdirectory in the backup space for that day, so that over time your backup space will look like that:
$BACKUP_PATH/ |-- host3/ | |-- 2007-09-02/ | |-- 2007-09-03/ | |-- 2007-09-05/ | |-- 2007-09-06/ | |-- 2007-09-07/ | |-- 2007-09-08/ | |-- 2007-09-09/ | |-- 2007-09-10/ | `-- 2007-09-11/ |-- host2/ | |-- 2007-09-02/ | |-- 2007-09-03/ | |-- 2007-09-05/ | |-- 2007-09-06/ | |-- 2007-09-07/ | |-- 2007-09-08/ | |-- 2007-09-09/ | |-- 2007-09-10/ | `-- 2007-09-11/ `-- host1/ |-- 2007-09-05/ |-- 2007-09-06/ |-- 2007-09-07/ |-- 2007-09-08/ |-- 2007-09-09/ |-- 2007-09-10/ `-- 2007-09-11/
Under each date subdirectory you will find a complete copy of the host from that day. Usually RBME is run from cron on a daily or nightly base to keep all hosts always backed up.
RBME produces a nice report of it's activities that serves as a daily backup report:
RSYNC BACKUP MADE EASY version 1.0 / 2007-09-12 Copyright (C) 2006 Schlomo Schapiro, pro business Berlin AG Licensed under the GNU General Public License, see http://www.gnu.org/licenses/gpl.html for full text Backing up to /media/backup (30032 MB & 15525831 inodes available) Free disk space requirements: 20000 MB & 100000 inodes before host backup 30000 MB & 200000 inodes after host backup Backup of 'host4:/' in host4/2007-09-11 against 2007-09-10 Backup runtime was 00:04:01 The backup for host4 utilizes 78 MB & 19836 inodes Removing old backups to free some disk space Oldest backup (from 9): 2007-08-31 Having 29954 MB, deleting /media/backup/host4/2007-08-31 in 116 seconds. Backup of 'host2:/ /home /var /media/install/sles10' in host2/2007-09-11 against 2007-09-10 Backup runtime was 00:55:12 The backup for host2 utilizes 1521 MB & 170852 inodes Removing old backups to free some disk space Oldest backup (from 98): 2007-05-13 Having 28635 MB, deleting /media/backup/host2/2007-05-13 in 1043 seconds. Having 29860 MB, deleting /media/backup/host2/2007-05-14 in 1309 seconds. Backup of 'host1:/' in host1/2007-09-11 against 2007-09-10 Backup runtime was 00:00:15 The backup for host1 utilizes 7 MB & 193 inodes Backup of 'host7:/' in host7/2007-09-11 against 2007-09-10 Backup runtime was 00:03:12 The backup for host7 utilizes 58 MB & 11059 inodes Backup of 'host5:/' in host5/2007-09-11 against 2007-09-10 Backup runtime was 00:01:20 The backup for host5 utilizes 40 MB & 5450 inodes Backup of 'host3:/' in host3/2007-09-11 against 2007-09-10 Backup runtime was 00:00:41 The backup for host3 utilizes 18 MB & 1230 inodes Backup of 'host6:/' in host6/2007-09-11 against 2007-09-10 Backup runtime was 00:03:19 The backup for host6 utilizes 154 MB & 6587 inodes Backup filesystem has now 30788 MB available Backup filesystem has now 16004755 inodes available Backup runtime was 01:49:34 Distribution of backup history (in % of backup amounts): ------------------------------------------------- 1.54 host1 16.69 host2 21.34 host3 1.54 host4 22.37 host5 19.44 host6 17.3 host7 ------------------------------------------------- Thank you for using RSYNC BACKUP MADE EASY
You should use the information about inode requirements to fine-tune the RBME configuration to guarantee sufficient free inodes before the backup starts. The amount of used MB and inodes gives an indication about the change rate of each host as only changed files (or new directories) occupy new inodes. Finally RBME gives a short statistical overview about the amount of old backups kept for each host.
RBME comes with a configuration file in /etc/rbme.conf where you should set your own values for the amount of free space or inodes that RBME should keep on your backup filesystem (hence better to use a dedicated filesystem). You can also set how many backups to always keep, even if it would mean violation the disk space thresholds. This ensures to always keep a minimum amount of old backups.
Specify the hosts and paths to backup as parameters to RBME. Two formats are recognized:
host'host:/ /var /boot /home'Excludes are managed locally on each backup host, to keep the information local to the host (and allow a local admin to manage his/her own excludes). Simply create a file /etc/backup-excludes.lst on each host to exclude files and directories. Please refer to the rsync manpage (section INCLUDE/EXCLUDE PATTERN RULES) for full details on the format. A small example will suffice here:
/media/stuff/backup/* /media/scratch/backup/* /media/files/backup/*
This example excludes various other backup directories on a server from this backup. The /* notation makes sure that the directory itself is backed up, but not the content.
To verify the configuration simply run RBME without a parameter:
# rbme RSYNC BACKUP MADE EASY version 1.2 / 2007-12-26 Copyright (C) 2006,2007 Schlomo Schapiro, probusiness Berlin AG Licensed under the GNU General Public License, see http://www.gnu.org/licenses/gpl.html for full text Usage: /home/temp/rbme <host:/path> ... host:/path is in rsync notation, see rsync(1) for more information Use host without :/path to specify entire host (with certain defaults excluded). Include / Exclude files: On each backup host, /etc/backup-excludes.lst can be used to specify local excludes (in rsync notation !) and /etc/backup-includes.lst can be used to specify local includes (though usually one uses only excludes, as the includes do NOT include files not part of the :/path spec !). Instance support: Create symlinks to the executable to create more instances with dedicated configuration. Each instance will have a dedicated logfile and produce different email reports. Current configuration: (change settings in /etc/rbme.conf (Master) and /etc/rbme.conf (Instance)) BACKUP_PATH = /media/backup MIN_FREE_BEFORE_HOST_BACKUP = 3000 MIN_INODES_BEFORE_HOST_BACKUP = 50000 MIN_FREE_AFTER_HOST_BACKUP = 10000 MIN_INODES_AFTER_HOST_BACKUP = 100000 MIN_KEEP_OLD_BACKUPS = 5 RSYNC = /usr/bin/rsync RSYNC_RSH = ssh RSYNC_EXTRA_OPTIONS = VERBOSE = yes REPORT = yes STATISTICS = yes DEBUG = MAILTO = MAILFROM = root@sles10-compile.home MAILSTYLE = all LOGFILE = /var/log/rbme.log.27 Project homepage: http://www.schapiro.org/schlomo/projects
RBME is distributed as tar.gz archive or as noarch RPM, which is the recommended installation method for RPM-based systems. After installing RBME you have to edit /etc/rbme.conf to adapt RBME to your environment (at least set the backup path).
Start from doing some test runs with a few hosts to get the feeling for RBME and then you can create a cron job that will run RBME regularly and mail the report back to you.
deb http://www.fbunet.de/debian/binary ./
send feedback to webmaster @ schapiro . org
