rsync-system-backup: Linux system backups powered by rsync

Welcome to the documentation of rsync-system-backup version 1.1!

User documentation

The readme explains the high level concepts and is mostly targeted at users of the command line interface. It’s probably the best place to start reading:

rsync-system-backup: Linux system backups powered by rsync

https://travis-ci.org/xolox/python-rsync-system-backup.svg?branch=master https://coveralls.io/repos/xolox/python-rsync-system-backup/badge.svg?branch=master

The rsync-system-backup program uses rsync to create full system backups of Linux systems. Supported backup destinations include local disks (possibly encrypted using LUKS) and remote systems that are running an SSH server or rsync daemon. Each backup produces a timestamped snapshot and these snapshots are rotated according to a rotation scheme that you can configure. The package is currently tested on cPython 2.7, 3.4, 3.5, 3.6, 3.7 and PyPy (2.7).

Status

While this project brings together more than ten years of experience in creating (system) backups using rsync, all of the actual Python code was written in the first few months of 2017 and has seen limited real world use. The project does however have an automated test suite with more than 90% test coverage and my intention is to extend the test coverage further.

In May 2018 I changed the status from alpha to beta as part of release 1.0. The bump in major version number was triggered by a backwards incompatible code change, however at that point I had been using rsync-system-backup to make local backups of several of my Linux systems for the majority of a year. Also several colleagues of mine have used the how-to on setting up unattended backups to an encrypted USB disk.

Warning

Please use the --dry-run option when you’re getting familiar with how rsync-system-backup works and don’t remove the option until you’re confident that you have the right command line, because using rsync-system-backup in the wrong way can cause data loss (for example by accidentally swapping the SOURCE and DESTINATION arguments).

Installation

The rsync-system-backup package is available on PyPI which means installation should be as simple as:

$ pip install rsync-system-backup

There’s actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).

Usage

There are two ways to use the rsync-system-backup package: As the command line program rsync-system-backup and as a Python API. For details about the Python API please refer to the API documentation available on Read the Docs. The command line interface is described below.

Command line

Usage: rsync-system-backup [OPTIONS] [SOURCE] DESTINATION

Use rsync to create full system backups.

The required DESTINATION argument specifies the (possibly remote) location where the backup is stored, in the syntax of rsync’s command line interface. The optional SOURCE argument defaults to ‘/’ which means the complete root filesystem will be included in the backup (other filesystems are excluded).

Please use the --dry-run option when getting familiar with this program and don’t remove it until you’re confident that you have the right command line, because using this program in the wrong way can cause data loss (for example by accidentally swapping the SOURCE and DESTINATION arguments).

Supported locations include:

  • Local disks (possibly encrypted using LUKS).
  • Remote systems that allow SSH connections.
  • Remote systems that are running an rsync daemon.
  • Connections to rsync daemons tunneled over SSH.

The backup process consists of several steps:

  1. First rsync is used to transfer all (relevant) files to a destination directory (whether on the local system or a remote system). Every time a backup is made, this same destination directory is updated.
  2. After the files have been transferred a ‘snapshot’ of the destination directory is taken and stored in a directory with a timestamp in its name. These snapshots are created using ‘cp --archive --link’.
  3. Finally the existing snapshots are rotated to purge old backups according to a rotation scheme that you can customize.

Supported options:

Option Description
-b, --backup Create a backup using rsync but don’t create a snapshot and don’t rotate old snapshots unless the --snapshot and/or --rotate options are also given.
-s, --snapshot

Create a snapshot of the destination directory but don’t create a backup and don’t rotate old snapshots unless the --backup and/or --rotate options are also given.

This option can be used to create snapshots of an rsync daemon module using a ‘post-xfer exec’ command. If DESTINATION isn’t given it defaults to the value of the environment variable $RSYNC_MODULE_PATH.
-r, --rotate

Rotate old snapshots but don’t create a backup and snapshot unless the --backup and/or --snapshot options are also given.

This option can be used to rotate old snapshots of an rsync daemon module using a ‘post-xfer exec’ command. If DESTINATION isn’t given it defaults to the value of the environment variable $RSYNC_MODULE_PATH.
-m, --mount=DIRECTORY

Automatically mount the filesystem to which backups are written.

When this option is given and DIRECTORY isn’t already mounted, the ‘mount’ command is used to mount the filesystem to which backups are written before the backup starts. When ‘mount’ was called before the backup started, ‘umount’ will be called when the backup finishes.

An entry for the mount point needs to be defined in /etc/fstab for this to work.
-c, --crypto=NAME

Automatically unlock the encrypted filesystem to which backups are written.

When this option is given and the NAME device isn’t already unlocked, the cryptdisks_start command is used to unlock the encrypted filesystem to which backups are written before the backup starts. When cryptdisks_start was called before the backup started, cryptdisks_stop will be called when the backup finishes.

An entry for the encrypted filesystem needs to be defined in /etc/crypttab for this to work. If the device of the encrypted filesystem is missing and rsync-system-backup is being run non-interactively, it will exit gracefully and not show any desktop notifications.

If you want the backup process to run fully unattended you can configure a key file in /etc/crypttab, otherwise you will be asked for the password each time the encrypted filesystem is unlocked.
-t, --tunnel=TUNNEL_SPEC Connect to an rsync daemon through an SSH tunnel. This provides encryption for rsync client to daemon connections that are not otherwise encrypted. The value of TUNNEL_SPEC is expected to be an SSH alias, host name or IP address. Optionally a username can be prefixed (followed by ‘@’) and/or a port number can be suffixed (preceded by ‘:’).
-i, --ionice=CLASS Use the ‘ionice’ program to set the I/O scheduling class and priority of the ‘rm’ invocations used to remove backups. CLASS is expected to be one of the values ‘idle’, ‘best-effort’ or ‘realtime’. Refer to the man page of the ‘ionice’ program for details about these values.
-u, --no-sudo By default backup and snapshot creation is performed with superuser privileges, to ensure that all files are readable and filesystem metadata is preserved. The -u, --no-sudo option disables the use of ‘sudo’ during these operations.
-n, --dry-run Don’t make any changes, just report what would be done. This doesn’t create a backup or snapshot but it does run rsync with the --dry-run option.
--multi-fs Allow rsync to cross filesystem boundaries. This option has the opposite effect of the rsync option --one-file-system because rsync-system-backup defaults to running rsync with --one-file-system and must be instructed not to using --multi-fs.
-x, --exclude=PATTERN Selectively exclude certain files from being included in the backup. Refer to the rsync documentation for allowed PATTERN syntax. Note that rsync-system-backup always uses the ‘rsync --one-file-system’ option.
-f, --force By default rsync-system-backup refuses to run on non-Linux systems because it was designed specifically for use on Linux. The use of the -f, --force option sidesteps this sanity check. Please note that you are on your own if things break!
--disable-notifications By default a desktop notification is shown (using notify-send) before the system backup starts and after the backup finishes. The use of this option disables the notifications (notify-send will not be called at all).
-v, --verbose Make more noise (increase logging verbosity). Can be repeated.
-q, --quiet Make less noise (decrease logging verbosity). Can be repeated.
-h, --help Show this message and exit.

How it works

I’ve been finetuning my approach to Linux system backups for years now and during that time rsync has become my swiss army knife of choice :-). I also believe that comprehensive documentation can be half the value of an open source project. The following sections attempt to provide a high level overview of my system backup strategy:

The (lack of) backup format

Each backup is a full copy of the filesystem tree, stored in the form of individual files and directories on the destination. This “backup format” makes it really easy to navigate through and recover from backups because you can use whatever method you are comfortable with, whether that is a file browser, terminal, Python script or even chroot :-).

Note

You may want to configure updatedb to exclude the directory containing your system backups, otherwise the locate database will grow enormously.

Rotation of snapshots

Snapshots can be rotated according to a flexible rotation scheme, for example I’ve configured my laptop backup rotation to preserve the most recent 24 hourly backups, 30 daily backups and endless monthly backups.

Backup destinations

While developing, maintaining and evolving backup scripts for various Linux laptops and servers I’ve learned that backups for different systems require different backup destinations and connection methods:

Encrypted USB disks

There’s a LUKS encrypted USB disk on my desk at work that I use to keep hourly, daily and monthly backups of my work laptop. The disk is connected through the same USB hub that also connects my keyboard and mouse so I can’t really forget about it :-).

Automatic mounting

Before the backup starts, the encrypted disk is automatically unlocked and mounted. The use of a key file enables this process to run unattended in the background. Once the backup is done the disk will be unmounted and locked again, so that it can be unplugged at any time (as long as a backup isn’t running of course).

Local server (rsync daemon)

My personal laptop transfers hourly backups to the rsync daemon running on the server in my home network using a direct TCP connection without SSH. Most of the time the laptop has an USB Ethernet adapter connected but the backup runs fine over a wireless connection as well.

Remote server (rsync daemon over SSH tunnel)

My VPS (virtual private server) transfers nightly backups to the rsync daemon running on the server in my home network over an SSH tunnel in order to encrypt the traffic and restrict access. The SSH account is configured to allow tunneling but disallow command execution. This setup enables the rsync client and server to run with root privileges without allowing the client to run arbitrary commands on the server.

Alternative connection methods

Backing up to a local disk limits the effectiveness of backups but using SSH access between systems gives you more than you bargained for, because you’re allowing arbitrary command execution. The rsync daemon provides an alternative that does not allow arbitrary command execution. The following sections discuss this option in more detail.

Using rsync daemon

To be able to write files as root and preserve all filesystem metadata, rsync must be running with root privileges. However most of my backups are stored on remote systems and opening up remote root access over SSH just to transfer backups feels like a very blunt way to solve the problem :-).

Fortunately another solution is available: Configure an rsync daemon on the destination and instruct your rsync client to connect to the rsync daemon instead of connecting to the remote system over SSH. The rsync daemon configuration can restrict the access of the rsync client so that it can only write to the directory that contains the backup tree.

In this setup no SSH connections are used and the traffic between the rsync client and server is not encrypted. If this is a problem for you then continue reading the next section.

Enabling rsync daemon

On Debian and derivatives like Ubuntu you can enable and configure an rsync daemon quite easily:

  1. Make sure that rsync is installed:

    $ sudo apt-get install rsync

  2. Enable the rsync daemon by editing /etc/default/rsync and changing the line RSYNC_ENABLE=false to RSYNC_ENABLE=true. Here’s a one liner that accomplishes the task:

    $ sudo sed -i 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsync

  3. Create the configuration file /etc/rsyncd.conf and define at least one module. Here’s an example based on my rsync daemon configuration:

    # Global settings.
max connections = 4
log file = /var/log/rsyncd.log

# Defaults for modules.
read only = no
uid = 0
gid = 0

# Daily backups of my VPS.
[vps_backups]
path = /mnt/backups/vps/latest
post-xfer exec = /usr/sbin/process-vps-backups

# Hourly backups of my personal laptop.
[laptop_backups]
path = /mnt/backups/laptop/latest
post-xfer exec = /usr/sbin/process-laptop-backups

    The post-xfer exec directives configure the rsync daemon to create a snapshot once the backup is done and rotate old snapshots afterwards.

  4. Once you’ve created /etc/rsyncd.conf you can start the rsync daemon:

    $ sudo service rsync start

  5. If you’re using a firewall you should make sure that the rsync daemon port is whitelisted to allow incoming connections. The rsync daemon port number defaults to 873. Here’s an iptables command to accomplish this:

    $ sudo iptables -A INPUT -p tcp -m tcp --dport 873 -m comment --comment "rsync daemon" -j ACCEPT
Tunneling rsync daemon connections

When your backups are transferred over the public internet you should definitely use SSH to encrypt the traffic, but if you’re at all security conscious then you probably won’t like having to open up remote root access over SSH just to transfer backups :-).

The alternative is to use a non privileged SSH account to set up an SSH tunnel that redirects network traffic to the rsync daemon. The login shell of the SSH account can be set to /usr/sbin/nologin (or something similar like /bin/false) to disable command execution, in this case you need to pass -N to the SSH client.

Contact

The latest version of rsync-system-backup is available on PyPI and GitHub. The documentation is hosted on Read the Docs and includes a changelog. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.

License

This software is licensed under the MIT license.

© 2019 Peter Odding.

The following instructions are also intended for users of the command line interface but are too detailed to be included in the readme:

How to set up unattended backups to an encrypted USB disk

This document explains how to set up unattended Linux system backups to an encrypted USB disk using LUKS filesystem encryption. These instructions are tested on Ubuntu (to be more specific I’ve used this process on 12.04, 14.04 and 16.04) but I’d expect them to work just as well on Debian and Linux distributions based on Debian.

Most of the steps in this how-to should in fact work fine on any Linux system (maybe with minor adjustments here and there) however there is one important thing to note: the configuration file /etc/crypttab and the commands cryptdisks_start and cryptdisks_stop are a Debian-ism that may not be available on other Linux distributions. The relevant sections explain how to tackle this (long story short: I wrote a fallback).

Install rsync-system-backup

There are several ways to install rsync-system-backup, for example:

# Make sure pip (the Python package manager) and related packages are installed.
sudo apt-get install python-{pip,pkg-resources,setuptools}

# Use pip to install the Python package we need in /usr/local. The
# executable will be available at /usr/local/bin/rsync-system-backup.
sudo pip install --upgrade rsync-system-backup

You can can also install the Python package and its dependencies in your home directory if you prefer that over “polluting” the system wide /usr/local directory:

# Use pip to install the Python package we need in ~/.local. The
# executable will be available at ~/.local/bin/rsync-system-backup.
pip install --upgrade --user rsync-system-backup

If that is still “too global” for your tastes then feel free to set up a Python virtual environment ;-).

Prepare the disk encryption

Create a key file

We will use a key file to enable rsync-system-backup to unlock the encrypted USB disk without requiring user interaction due to a password prompt. Basically the contents of the key file will serve as an alternate password that can be used in a noninteractive setting.

# Create a directory to store the key file.
sudo mkdir -p /root/keys

# Generate the key file from two kilobytes of pseudorandom data.
sudo dd if=/dev/urandom of=/root/keys/backups.key bs=512 count=4

# Make sure the directory and key file are private to the root user.
sudo chown -R root:root /root/keys
sudo chmod u=rwx,go= /root/keys
sudo chmod u=r,go= /root/keys/backups.key

Warning

I’m assuming here that the computer that you want to create backups of already has full disk encryption. If this is not the case it means that anyone with physical access to the computer can just power it off, rip out the hard disk and extract the contents of /root/keys/backups.key, compromising the security of your backups!

Enable encryption on the USB disk

Enabling encryption on the USB disk will effectively wipe the existing contents of the disk, so you need to make sure of two things:

  1. The existing contents of the disk have been backed up.
  2. You’re specifying the correct device file in the following command (please triple check or you might wipe the wrong disk).
# Enable LUKS disk encryption on the USB disk.
sudo cryptsetup luksFormat /dev/sdx /root/keys/backups.key

In the command above /dev/sdx is the device file (this is what you need to change, see figuring out the correct device file for hints) and /root/keys/backups.key is the name of the key file that we created in the previous step.

Careful readers will notice that I’m not bothering to create a partition table on the USB disk, that’s because we don’t need it :-).

Make sure the disk is not in use

The luksFormat command above may give you an error like:

Cannot format device /dev/sdx which is still in use

If this happens then most likely the USB disk that you attached already has a filesystem on it and your desktop environment automatically mounted that filesystem. You will need to unmount that filesystem before you can enable encryption on the disk. If you don’t know how to do that:

  1. Follow the steps in the section figuring out the correct device file and take note of the device file corresponding to the USB disk.

  2. Run the mount command to get a list of mounted filesystems and look for lines that mention the relevant device file. Most likely a number will be appended at the end of the device file (this indicates a partition on the USB disk).

  3. For each of the relevant entries in the mount output, run the following command:

    sudo umount /dev/sdx1

    In the command above /dev/sdx1 is the device file of a partition on the USB disk (this is what you need to change).

Figuring out the correct device file

If you don’t know or you’re not sure what the device file for the luksFormat command above should be, here’s one relatively foolproof way to figure it out:

  1. Disconnect the USB disk from your computer.

  2. Open a terminal and use the following command to observe new log entries being added to the system log:

    # Follow the system log (watch for new entries).
sudo tail -fn 0 /var/log/syslog

  3. Connect the USB disk to your computer and give the disk a few seconds to spin up and properly establish a USB connection to your computer.

  4. Observe the entries that just appeared in the system log. If you study them carefully you should be able to figure out the name of the device file.

Configure a recovery password

If your computer’s hard disk breaks or your computer is stolen you will lose the key file required to unlock your encrypted backups, which would be rather ironic but not in a fun way :-P. To avoid this situation we can configure the disk encryption with a recovery password:

# Configure a recovery password.
sudo cryptsetup --key-file=/root/keys/backups.key luksAddKey /dev/sdx

In the command above /dev/sdx is the device file, this should be the same device file you used in the previous step.

Configure the encrypted disk

Once encryption has been enabled we can configure the encrypted disk in /etc/crypttab. To do so we first need to determine the unique identifier of the encrypted disk:

# Determine the UUID of the encrypted disk.
sudo blkid /dev/sdx

In the command above /dev/sdx is the device file, this should be the same device file you used in the previous step. The blkid command will output a string called a UUID (a universally unique identifier), you need to copy this to your clipboard (or have photographic memory). Now that we know the UUID we can add the /etc/crypttab entry:

# Use a text editor to configure the encrypted disk.
sudo nano /etc/crypttab

If the file doesn’t exist yet it implies that you’re not using full disk encryption on your computer. Please reconsider! But I digress. Now you need to add a new line to the file, something like this:

backups UUID=13f6e17e-8c8b-4009-a7b3-356992415141 /root/keys/backups.key luks,discard,noauto

Replace the part after UUID= with the output of blkid. Everything else should be fine as is, unless you’ve chosen a different location for the key file.

Unlock the encrypted disk

Thanks to the /etc/crypttab entry that we added in the previous step, unlocking the disk is very simple:

# Unlock the encrypted backups disk.
sudo cryptdisks_start backups

This won’t ask for a password because we configured a key file. If you get a “command not found” error then here are two suggestions:

  1. If you’re running a Linux distribution based on Debian (like Ubuntu) then you can install cryptdisks_start and cryptdisks_stop as follows:

    # Make sure `cryptdisks_start' and `cryptdisks_stop' are installed.
sudo apt-get install cryptsetup

  2. If you’re running a Linux distribution that’s not based on Debian then the cryptdisks_start and cryptdisks_stop programs may not be available to you. Don’t worry though, I’ve got your back! ;-)

    Because we’ve already installed rsync-system-backup its dependencies are also available. One of these dependencies installs the following two command line programs:

    • cryptdisks-start-fallback
    • cryptdisks-stop-fallback

    These programs are not as full featured as their “official” counterparts but they should work fine for the purposes of this how-to. Instead of the command given at the start of this section, please use the following command:

    sudo cryptdisks-start-fallback backups

Prepare the encrypted filesystem

Format the encrypted filesystem

After the encrypted disk is unlocked using cryptdisks_start the device file /dev/mapper/backups provides access to the unlocked data. Encrypting the disk hasn’t created an actual filesystem yet so that’s what we’ll do next:

sudo mkfs.ext4 /dev/mapper/backups
Configure the encrypted filesystem

We’ll add an entry to /etc/fstab so that it’s as easy to mount the filesystem as it was easy to unlock the disk:

# Use a text editor to configure the encrypted filesystem.
sudo nano /etc/fstab

Add a new line to the file, something like this:

/dev/mapper/backups /mnt/backups ext4 noauto,errors=remount-ro 0 0

Also make sure the mount point exists:

sudo mkdir -p /mnt/backups
Mount the encrypted filesystem

This should be familiar to most of you:

sudo mount /mnt/backups
Decide on a directory layout

On my backup disks I am using a directory layout of multiple levels because my backups and I go way back :-). The first level consists of the names I chose to describe the laptops I’ve had over the years:

  • /mnt/backups
    • zenbook
    • hp-probook
    • macbook-pro

Each of these directories has subdirectories with the names of the Ubuntu releases that were installed on those laptops over the years:

  • /mnt/backups
    • zenbook
      • lucid
      • precise
    • hp-probook
      • precise
      • trusty
    • macbook-pro
      • xenial

Each of the directories named after an Ubuntu release stores a collection of timestamped system backups, something like this:

  • /mnt/backups
    • zenbook
      • lucid
        • 2011-02-05 15:30
        • 2011-03-19 11:45
      • precise
        • 2013-04-10 14:00
        • 2013-05-10 14:00
    • hp-probook
      • precise
        • 2014-03-12 16:15
      • trusty
        • 2016-06-15 12:00
    • macbook-pro
      • xenial
        • 2017-03-19 23:15
        • 2017-04-01 12:34
        • 2017-05-02 17:00
        • latest

The dates were made up and in reality I have hundreds of timestamped system backups, but you get the idea :-).

Whether you use the same directory layout or something simpler is up to you.

Create your first backup

Here’s an example of how you can create a system backup:

sudo rsync-system-backup -c backups -m /mnt/backups /mnt/backups/latest

That last directory must be a subdirectory of /mnt/backups, if you want to keep things simple then just use /mnt/backups/latest (whatever you do, don’t just pass it /mnt/backups).

If you get a “command not found” error from sudo try the following instead:

sudo $(which rsync-system-backup) -c backups -m /mnt/backups /mnt/backups/latest

Configure unattended backups

The final part of this how-to configures your system to automatically run rsync-system-backup at an interval of your choosing, for example once every four hours. The easiest way to accomplish this is using cron. To do so we’ll create a new configuration file:

# Use a text editor to configure unattended backups.
sudo nano /etc/cron.d/rsync-system-backup

Create the file with the following contents:

# Cron by default starts subcommands in a very sparse environment where the
# $PATH contains just /usr/bin and /bin. Since we expect a reasonably sane
# $PATH we have to set it ourselves:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Create a full system backup every four hours.
0 */4 * * * root rsync-system-backup -c backups -m /mnt/backups /mnt/backups/latest

Depending on how you installed rsync-system-backup you may need to adjust the PATH variable or change the program name into an absolute pathname.

Silencing desktop notifications

When rsync-system-backup is running non-interactively and it finds that the device of the encrypted filesystem is missing, it will exit gracefully and not show any desktop notifications. This is intended to avoid noise when the backup disk isn’t connected.

If the desktop notifications announcing the start and completion of a system backup drive you bonkers, add the --disable-notifications option to the rsync-system-backup command line to silence desktop notifications.

API documentation

The following documentation is targeted at people who are interested in using the Python API:

API documentation

The following documentation is based on the source code of version 1.1 of the rsync-system-backup package:

rsync_system_backup

Simple to use Python API for Linux system backups powered by rsync.

The rsync_system_backup module contains the Python API of the rsync-system-backup package. The core logic of the package is contained in the RsyncSystemBackup class.

rsync_system_backup.DEFAULT_ROTATION_SCHEME = {'daily': 7, 'hourly': 24, 'monthly': 'always', 'weekly': 4}

The default rotation scheme for system backup snapshots (a dictionary).

class rsync_system_backup.RsyncSystemBackup(**kw)[source]

Python API for the rsync-system-backup program.

The execute() method is the main entry point. If you’re looking for finer grained control refer to unlock_device(), mount_filesystem(), transfer_changes(), create_snapshot() and rotate_snapshots().

When you initialize a RsyncSystemBackup object you are required to provide a value for the destination property. You can set the values of the backup_enabled, crypto_device, destination, dry_run, exclude_list, excluded_roots, force, ionice, mount_point, multi_fs, notifications_enabled, rotate_enabled, rotation_scheme, snapshot_enabled, source, source_context and sudo_enabled properties by passing keyword arguments to the class initializer.

Here’s an overview of the RsyncSystemBackup class:

Superclass: PropertyManager
Public methods: create_snapshot(), ensure_supported_platform(), execute(), execute_helper(), mount_filesystem(), notify_failed(), notify_finished(), notify_starting(), rotate_snapshots(), transfer_changes() and unlock_device()
Properties: backup_enabled, crypto_device, crypto_device_available, crypto_device_unlocked, crypttab_entry, destination, destination_context, dry_run, exclude_list, excluded_roots, force, ionice, mount_point, mount_point_active, multi_fs, notifications_enabled, rotate_enabled, rotation_scheme, snapshot_enabled, source, source_context and sudo_enabled
backup_enabled[source]

True to enable transfer_changes(), False otherwise.

Note

The backup_enabled property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

crypto_device[source]

The name of the encrypted filesystem to use (a string or None).

Note

The crypto_device property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

crypto_device_available

True if the encrypted filesystem is available, False otherwise.

This property is an alias for the is_available property of crypttab_entry.

crypto_device_unlocked

True if the encrypted filesystem is unlocked, False otherwise.

This property is an alias for the is_unlocked property of crypttab_entry.

crypttab_entry[source]

The entry in /etc/crypttab corresponding to crypto_device.

The value of this property is computed automatically by parsing /etc/crypttab and looking for an entry whose target (the first of the four fields) matches crypto_device.

When an entry is found an EncryptedFileSystemEntry object is constructed, otherwise the result is None.

Note

The crypttab_entry property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

destination[source]

The destination where backups are stored (a Destination object).

The value of destination defaults to the value of the environment variable $RSYNC_MODULE_PATH which is set by the rsync daemon before it runs the post-xfer exec command.

Note

The destination property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named destination (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

destination_context[source]

The execution context of the system that stores the backup (the destination).

This is an execution context created by executor.contexts.

Raises:DestinationContextUnavailable when the destination is an rsync daemon module (which doesn’t allow arbitrary command execution).

Note

The destination_context property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

dry_run[source]

True to simulate the backup without writing any files, False otherwise.

Note

The dry_run property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

multi_fs[source]

True to allow rsync to cross filesystem boundaries, False otherwise.

This property has the opposite effect of the rsync command line option --one-file-system because multi_fs defaults to False which means rsync is run with --one-file-system. You can set multi_fs to True to omit --one-file-system from the rsync command line.

Note

The multi_fs property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

exclude_list[source]

A list of patterns (strings) that are excluded from the system backup.

The patterns in exclude_list are passed on to rsync using the --exclude option.

Note

The exclude_list property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.

excluded_roots[source]

A list of patterns (strings) that are excluded from the system backup.

All of the patterns in this list will be rooted to the top of the filesystem hierarchy when they’re given the rsync, to avoid unintentionally excluding deeply nested directories that happen to match names in this list. This is done using the --filter=-/ PATTERN option.

Note

The excluded_roots property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.

force[source]

True to run rsync-system-backup on unsupported platforms, False otherwise.

Note

The force property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ionice[source]

The I/O scheduling class for rsync (a string or None).

When this property is set ionice will be used to set the I/O scheduling class for rsync. This can be useful to reduce the impact of backups on the rest of the system.

The value of this property is expected to be one of the strings ‘idle’, ‘best-effort’ or ‘realtime’.

Note

The ionice property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

mount_point[source]

The pathname of the mount point to use (a string or None).

Note

The mount_point property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

mount_point_active

True if mount_point is mounted already, False otherwise.

notifications_enabled[source]

Whether desktop notifications are used (a boolean).

By default desktop notifications are enabled when a real backup is being made but disabled during dry runs.

Note

The notifications_enabled property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

rotation_scheme[source]

The rotation scheme for snapshots (a dictionary, defaults to DEFAULT_ROTATION_SCHEME).

Note

The rotation_scheme property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

snapshot_enabled[source]

True to enable create_snapshot(), False otherwise.

Note

The snapshot_enabled property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

source[source]

The pathname of the directory to backup (a string, defaults to ‘/’).

Note

The source property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

source_context[source]

The execution context of the system that is being backed up (the source).

This is expected to be an execution context created by executor.contexts. It defaults to executor.contexts.LocalContext.

Note

The source_context property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.

rotate_enabled[source]

True to enable rotate_snapshots(), False otherwise.

Note

The rotate_enabled property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

sudo_enabled[source]

True to run rsync and snapshot creation with superuser privileges, False otherwise.

Note

The sudo_enabled property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

execute()[source]

Execute the requested actions (backup, snapshot and/or rotate).

The execute() method defines the high level control flow of the backup / snapshot / rotation process according to the caller’s requested configuration:

  1. When backup_enabled is set notify_starting() shows a desktop notification to give the user a heads up that a system backup is about to start (because the backup may have a noticeable impact on system performance).
  2. When crypto_device is set unlock_device() ensures that the configured encrypted device is unlocked.
  3. When mount_point is set mount_filesystem() ensures that the configured filesystem is mounted.
  4. When backup_enabled is set transfer_changes() creates or updates the system backup on destination using rsync.
  5. When snapshot_enabled is set create_snapshot() creates a snapshot of the destination directory.
  6. When rotate_enabled is set rotate_snapshots() rotates snapshots.
  7. When backup_enabled is set notify_finished() shows a desktop notification to give the user a heads up that the system backup has finished (or failed).
ensure_supported_platform()[source]

Make sure we’re running on a supported platform.

Raises:UnsupportedPlatformError when the output of the uname command doesn’t include the word ‘Linux’ and force is False.

When force is True this method logs a warning message instead of raising an exception.

execute_helper()[source]

Helper for execute().

notify_starting()[source]

Notify the desktop environment that a system backup is starting.

notify_finished(timer)[source]

Notify the desktop environment that a system backup has finished.

notify_failed(timer)[source]

Notify the desktop environment that a system backup has failed.

unlock_device()[source]

Automatically unlock the encrypted filesystem to which backups are written.

Raises:

The following exceptions can be raised:

When crypto_device is set this method uses cryptdisks_start() to unlock the encrypted filesystem to which backups are written before the backup starts. When cryptdisks_start() was called before the backup started, cryptdisks_stop() will be called when the backup finishes.

To enable the use of cryptdisks_start() and cryptdisks_stop() you need to create an /etc/crypttab entry that maps your physical device to a symbolic name. If you want this process to run fully unattended you can configure a key file in /etc/crypttab, otherwise you will be asked for the password when the encrypted filesystem is unlocked.

mount_filesystem()[source]

Automatically mount the filesystem to which backups are written.

Raises:

The following exceptions can be raised:

When mount_point is set this method uses the mount command to mount the filesystem to which backups are written before the backup starts. When mount was called before the backup started, umount will be called when the backup finishes. An entry for the mount point needs to be defined in /etc/fstab.

transfer_changes()[source]

Use rsync to synchronize the files on the local system to the backup destination.

Raises:InvalidDestinationDirectory when mount_point is set and destination is a local directory that is not located under mount_point.
create_snapshot()[source]

Create a snapshot of the destination directory.

Raises:

The following exceptions can be raised:
rotate_snapshots()[source]

Rotate system backup snapshots using rotate_backups.

Raises:

The following exceptions can be raised:

The values of the dry_run, ionice and rotation_scheme properties are passed on to the RotateBackups class.

rsync_system_backup.ensure_trailing_slash(expression)[source]

Add a trailing slash to rsync source/destination locations.

Parameters:expression – The rsync source/destination expression (a string).
Returns:The same expression with exactly one trailing slash.

rsync_system_backup.cli

Usage: rsync-system-backup [OPTIONS] [SOURCE] DESTINATION

Use rsync to create full system backups.

The required DESTINATION argument specifies the (possibly remote) location where the backup is stored, in the syntax of rsync’s command line interface. The optional SOURCE argument defaults to ‘/’ which means the complete root filesystem will be included in the backup (other filesystems are excluded).

Please use the --dry-run option when getting familiar with this program and don’t remove it until you’re confident that you have the right command line, because using this program in the wrong way can cause data loss (for example by accidentally swapping the SOURCE and DESTINATION arguments).

Supported locations include:

  • Local disks (possibly encrypted using LUKS).
  • Remote systems that allow SSH connections.
  • Remote systems that are running an rsync daemon.
  • Connections to rsync daemons tunneled over SSH.

The backup process consists of several steps:

  1. First rsync is used to transfer all (relevant) files to a destination directory (whether on the local system or a remote system). Every time a backup is made, this same destination directory is updated.
  2. After the files have been transferred a ‘snapshot’ of the destination directory is taken and stored in a directory with a timestamp in its name. These snapshots are created using ‘cp --archive --link’.
  3. Finally the existing snapshots are rotated to purge old backups according to a rotation scheme that you can customize.

Supported options:

Option Description
-b, --backup Create a backup using rsync but don’t create a snapshot and don’t rotate old snapshots unless the --snapshot and/or --rotate options are also given.
-s, --snapshot

Create a snapshot of the destination directory but don’t create a backup and don’t rotate old snapshots unless the --backup and/or --rotate options are also given.

This option can be used to create snapshots of an rsync daemon module using a ‘post-xfer exec’ command. If DESTINATION isn’t given it defaults to the value of the environment variable $RSYNC_MODULE_PATH.
-r, --rotate

Rotate old snapshots but don’t create a backup and snapshot unless the --backup and/or --snapshot options are also given.

This option can be used to rotate old snapshots of an rsync daemon module using a ‘post-xfer exec’ command. If DESTINATION isn’t given it defaults to the value of the environment variable $RSYNC_MODULE_PATH.
-m, --mount=DIRECTORY

Automatically mount the filesystem to which backups are written.

When this option is given and DIRECTORY isn’t already mounted, the ‘mount’ command is used to mount the filesystem to which backups are written before the backup starts. When ‘mount’ was called before the backup started, ‘umount’ will be called when the backup finishes.

An entry for the mount point needs to be defined in /etc/fstab for this to work.
-c, --crypto=NAME

Automatically unlock the encrypted filesystem to which backups are written.

When this option is given and the NAME device isn’t already unlocked, the cryptdisks_start command is used to unlock the encrypted filesystem to which backups are written before the backup starts. When cryptdisks_start was called before the backup started, cryptdisks_stop will be called when the backup finishes.

An entry for the encrypted filesystem needs to be defined in /etc/crypttab for this to work. If the device of the encrypted filesystem is missing and rsync-system-backup is being run non-interactively, it will exit gracefully and not show any desktop notifications.

If you want the backup process to run fully unattended you can configure a key file in /etc/crypttab, otherwise you will be asked for the password each time the encrypted filesystem is unlocked.
-t, --tunnel=TUNNEL_SPEC Connect to an rsync daemon through an SSH tunnel. This provides encryption for rsync client to daemon connections that are not otherwise encrypted. The value of TUNNEL_SPEC is expected to be an SSH alias, host name or IP address. Optionally a username can be prefixed (followed by ‘@’) and/or a port number can be suffixed (preceded by ‘:’).
-i, --ionice=CLASS Use the ‘ionice’ program to set the I/O scheduling class and priority of the ‘rm’ invocations used to remove backups. CLASS is expected to be one of the values ‘idle’, ‘best-effort’ or ‘realtime’. Refer to the man page of the ‘ionice’ program for details about these values.
-u, --no-sudo By default backup and snapshot creation is performed with superuser privileges, to ensure that all files are readable and filesystem metadata is preserved. The -u, --no-sudo option disables the use of ‘sudo’ during these operations.
-n, --dry-run Don’t make any changes, just report what would be done. This doesn’t create a backup or snapshot but it does run rsync with the --dry-run option.
--multi-fs Allow rsync to cross filesystem boundaries. This option has the opposite effect of the rsync option --one-file-system because rsync-system-backup defaults to running rsync with --one-file-system and must be instructed not to using --multi-fs.
-x, --exclude=PATTERN Selectively exclude certain files from being included in the backup. Refer to the rsync documentation for allowed PATTERN syntax. Note that rsync-system-backup always uses the ‘rsync --one-file-system’ option.
-f, --force By default rsync-system-backup refuses to run on non-Linux systems because it was designed specifically for use on Linux. The use of the -f, --force option sidesteps this sanity check. Please note that you are on your own if things break!
--disable-notifications By default a desktop notification is shown (using notify-send) before the system backup starts and after the backup finishes. The use of this option disables the notifications (notify-send will not be called at all).
-v, --verbose Make more noise (increase logging verbosity). Can be repeated.
-q, --quiet Make less noise (decrease logging verbosity). Can be repeated.
-h, --help Show this message and exit.
rsync_system_backup.cli.main()[source]

Command line interface for the rsync-system-backup program.

rsync_system_backup.cli.enable_explicit_action(options, explicit_action)[source]

Explicitly enable an action and disable other implicit actions.

Parameters:
  • options – A dictionary of options.
  • explicit_action – The action to enable (one of the strings ‘backup_enabled’, ‘snapshot_enabled’, ‘rotate_enabled’).

rsync_system_backup.destinations

Parsing of rsync destination syntax (and then some).

rsync_system_backup.destinations.RSYNCD_PORT = 873

The default port of the rsync daemon (an integer).

rsync_system_backup.destinations.LOCAL_DESTINATION = <_sre.SRE_Pattern object>

A compiled regular expression pattern to parse local destinations, used as a fall back because it matches any nonempty string.

rsync_system_backup.destinations.SSH_DESTINATION = <_sre.SRE_Pattern object>

A compiled regular expression pattern to parse remote destinations of the form [USER@]HOST:DEST (using an SSH connection).

rsync_system_backup.destinations.SIMPLE_DAEMON_DESTINATION = <_sre.SRE_Pattern object>

A compiled regular expression pattern to parse remote destinations of the form [USER@]HOST::MODULE[/DIRECTORY] (using an rsync daemon connection).

rsync_system_backup.destinations.ADVANCED_DAEMON_DESTINATION = <_sre.SRE_Pattern object>

A compiled regular expression pattern to parse remote destinations of the form rsync://[USER@]HOST[:PORT]/MODULE[/DIRECTORY] (using an rsync daemon connection).

rsync_system_backup.destinations.DESTINATION_PATTERNS = [<_sre.SRE_Pattern object>, <_sre.SRE_Pattern object>, <_sre.SRE_Pattern object>, <_sre.SRE_Pattern object>]

A list of compiled regular expression patterns to match destination expressions. The patterns are ordered by decreasing specificity.

class rsync_system_backup.destinations.Destination(**kw)[source]

The Destination class represents a location where backups are stored.

The expression property is a required property whose value is parsed to populate the values of the username, hostname, port_number, module and directory properties.

When you read the value of the expression property you get back a computed value based on the values of the previously mentioned properties. This makes it possible to manipulate the destination before passing it on to rsync.

When you initialize a Destination object you are required to provide a value for the expression property. You can set the values of the directory, expression, hostname, module, parent_directory, port_number, ssh_tunnel and username properties by passing keyword arguments to the class initializer.

Here’s an overview of the Destination class:

Superclass: PropertyManager
Special methods: __enter__() and __exit__()
Properties: directory, expression, hostname, module, parent_directory, port_number, ssh_tunnel and username
expression[source]

The destination in rsync’s command line syntax (a string).

Raises:InvalidDestinationError when you try to set this property to a value that cannot be parsed.

Note

The expression property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named expression (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

directory[source]

The pathname of the directory where the backup should be written (a string).

Note

The directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

hostname[source]

The host name or IP address of a remote system (a string).

Note

The hostname property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

module[source]

The name of a module exported by an rsync daemon (a string).

Note

The module property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

parent_directory[source]

The pathname of the parent directory of the backup directory (a string).

Raises:ParentDirectoryUnavailable when the parent directory can’t be determined because directory is empty or ‘/’.

Note

The parent_directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

port_number[source]

The port number of a remote rsync daemon (a number).

When ssh_tunnel is set the value of port_number defaults to executor.ssh.client.SecureTunnel.local_port, otherwise it defaults to RSYNCD_PORT.

Note

The port_number property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ssh_tunnel[source]

A SecureTunnel object or None (defaults to None).

Note

The ssh_tunnel property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

username[source]

The username for connecting to a remote system (a string).

Note

The username property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

__enter__()[source]

Automatically open ssh_tunnel when required.

__exit__(exc_type=None, exc_value=None, traceback=None)[source]

Automatically close ssh_tunnel when required

rsync_system_backup.exceptions

Custom exceptions used by rsync-system-backup.

exception rsync_system_backup.exceptions.RsyncSystemBackupError[source]

Base exception for custom exceptions raised by rsync-system-backup.

exception rsync_system_backup.exceptions.UnsupportedPlatformError[source]

Raised when an unsupported (non-Linux) platform is detected.

exception rsync_system_backup.exceptions.InvalidDestinationError[source]

Raised when the given destination expression can’t be parsed.

exception rsync_system_backup.exceptions.MissingBackupDiskError[source]

Raised when the encrypted filesystem isn’t available.

exception rsync_system_backup.exceptions.FailedToUnlockError[source]

Raised when cryptdisks_start fails to unlock the encrypted device.

exception rsync_system_backup.exceptions.FailedToMountError[source]

Raised when mount fails to mount the backup destination.

exception rsync_system_backup.exceptions.DestinationContextUnavailable[source]

Raised when snapshot creation and rotation are disabled because we’re connected to an rsync daemon.

exception rsync_system_backup.exceptions.ParentDirectoryUnavailable[source]

Raised when the parent directory of the backup directory cannot be determined.

exception rsync_system_backup.exceptions.InvalidDestinationDirectory[source]

Raised when the backup directory isn’t located inside the given mount point.

Change log

The change log lists notable changes to the project:

Changelog

The purpose of this document is to list all of the notable changes to this project. The format was inspired by Keep a Changelog. This project adheres to semantic versioning.

Release 1.1 (2019-08-02)

  • The --multi-fs option suggested in pull request #3 was added.

    This option has the opposite effect of the rsync option --one-file-system because rsync-system-backup defaults to running rsync with --one-file-system and must be instructed not to using --multi-fs.”

    This change was merged by cherry picking the relevant commit from the remote branch, to avoid pulling in unrelated changes (see the pull request for details).

  • Stop testing on Python 2.6 (because Travis CI no longer supports it and working around this would cost an unreasonable amount of time) and start testing support for Python 3.7.

Release 1.0 (2018-05-04)

Bug fix: SSH tunnel support actually works now :-P (backwards incompatible).

This week I switched the backups of my VPS over to rsync-system-backup. The biggest hurdle was the fact that I never finished nor tested support for SSH tunnels in rsync-system-backup which I needed now:

  • The command line interface didn’t expose the functionality.
  • Due to various bugs it wouldn’t have worked even if the functionality had been exposed.

The changes in this release:

  • Added the -t, --tunnel command line option.
  • Integrated SSH tunnel support provided by executor 19.3.
  • Removed the ForwardedDestination class (this functionality has been replaced by the new Destination.ssh_tunnel property).

Because the removal of ForwardedDestination is backwards incompatible I’ve decided to bump the major version number. It’s actually kind of fitting because I’ve been using rsync-system-backup for local backups for months now and that’s working fine; the main thing missing was indeed SSH tunnel support :-).

Release 0.11 (2018-04-30)

  • Added support for the -x, --exclude option.
  • Documented that --one-file-system is always used.

Release 0.10 (2018-04-30)

  • Switched to the more user friendly getopt.gnu_getopt().
  • Added this changelog, restructured the online documentation.
  • Documented the -f, --force option in the readme.
  • Integrated the use of property_manager.sphinx.

Release 0.9 (2017-07-11)

Explicitly handle unsupported platforms (by refusing to run without the -f, --force option). Refer to issue #1 for more information.

Release 0.8 (2017-06-24)

Don’t raise an exception when notify-send fails to deliver a desktop notification.

Release 0.7 (2017-06-23)

Ensure the destination directory is located under the expected mount point.

Release 0.6 (2017-06-23)

Incorporated the cryptdisks_start and cryptdisks_stop fallbacks into the how-to.

Release 0.5 (2017-06-21)

Gain independence from cryptdisks_start and cryptdisks_stop (a Debian-ism).

Release 0.4 (2017-06-21)

  • Gracefully handle missing backup disk.
  • Added a how-to to the documentation.

Release 0.3 (2017-06-06)

Made it possible to disable desktop notifications.

Release 0.2 (2017-05-06)

  • Don’t render a traceback on known errors.
  • Fixed broken usage message formatting.
  • Document Python 3.6 compatibility.
  • Changed Sphinx theme.

Release 0.1.1 (2017-04-17)

Changed system logging verbosity level from DEBUG to INFO.

Release 0.1 (2017-04-14)

Initial release (0.1, alpha).