Using btrfs on Linux to store raw image files in an efficient manner

Currently UrBackup is storing images of volumes as VHD (Virtual HardDisk) files. In UrBackup Server 1.4 the VHD file format was augmented with a custom compression. This combination gives following advantages over storing images as „raw“ files:

  1. Only used parts of the volumes are stored to the VHD files (sparse image)
  2. If the compression is enabled, the volume data is compressed
  3. Incremental image backups: A VHD file can reference another VHD file, so unchanged data does not have to be stored twice (differential image)

All this can be implemented with btrfs. Volume images can then be stored as raw files:

  1. There is a Linux system call to „punch holes“ into a file. Btrfs supports this. Using hole punching only the parts of the raw image file which are used are stored
  2. Btrfs has a compression feature
  3. You can copy a file on btrfs without copying the data the files contain. You can do this in a Linux console via „cp –reflink=always“. A file copied this way only adds another reference to the file data. The file only becomes a “real” copy at changed parts of the file (Copy on Write).

An incremental image backup can then be done like this: Create reflink of the last image backup, change the parts in the image file that have changed since the last image backup and then punch holes into the file to remove unused parts.

As you can see, this moves the implementation of compression, sparse image support and differential image support into btrfs, which is a good thing, because btrfs is starting to get widespread adoption.

The implementation using btrfs has several advantages:

  • The volumes are stored as raw files, which makes inter-operation easier. For example you can directly map the file to a Linux volume device, which makes the file look like a volume on a hard drive.
  • UrBackup can delete any image in the incremental backup chain and btrfs takes care of cleaning up the unused blocks. This makes running infinite incremental image backups viable. After the initial full backup you only need to make incremental image backups

The only disadvantage I saw till now is that UrBraw-copy-on-write-fileackup cannot accurately display the size of incremental image backups anymore. It always shows the size as if it was an uncompressed full image.

This new image file format will be available in UrBackup 1.5 if UrBackup is configured such that it uses btrfs.

Performance considerations for larger UrBackup server instances

If you are planning on setting up a larger UrBackup server instance you will find some hints about performance in this post. “Large” in this context is difficult to define, because it depends on the number of clients, the number of files and file sizes in the backups and the backup intervals.

If you plan on setting up a “larger” UrBackup instance you should keep the following things in mind:

  • UrBackup uses an internal (SQLite) database. This database could cause performance problems, especially with a large number of backed-up files and full file backups. The database should be stored on storage suited for databases.
  • UrBackup has some tuning options, but is nevertheless pretty optimized per default. You should only have to tune UrBackup in special circumstances. You will find information about the tuning options in the administration manual.
  • There are many platform options and each has its own considerations, so you should read up on the platform-specific performance considerations. For example you should not run FreeNAS virtualized.

Your system will almost certainly be IO limited. If you have a running system you can verify that by looking at the performance monitor on Windows, iostat on Linux and zfs iostat on FreeBSD. Often it is limited by random read/write performance (Input/Output operations per second).

If you want maximum IO performance following should therefore be the case:

  • The UrBackup database should be on an SSD. This should be a no-brainer, as this database does not get too large and SSDs are way faster than spinning disks. The random reads/writes are for example 900 times faster with a Samsung 840 Pro (97K IOPS).
  • The UrBackup database should not be on the same disk as the backup storage.
  • The UrBackup database should not be on a RAID-5, as this is not optimal for databases.
  • If the database is still the bottleneck (because it is a separate device you can find this out using iostat or an equivalent), you can use the “file entry cache” (see manual). This cache should be on a separate SSD, otherwise it will only cause more IO on the one device.
  • Save the filesystem metadata of your backup storage on an SSD and only the actual data on a spinning disk RAID-5/6 to get the maximum performance. This is only possible with btrfs on Linux.
  • Avoid full file backups. When doing a full file backup UrBackup has to load all files, calculate their hash value and look this value up in the database. This incurs a lot of IO on both the database and the backup storage (and the client). UrBackup can run an infinite amount of incremental file backups without any full file backups.
  • Optimize the maximum number of concurrent backups such that the throughput is maximized.

Ways to absolutely kill performance:

  • Save the UrBackup database on ZFS/btrfs on a spinning disk. Databases (those who use WAL or intent logging; including SQLite3) on ZFS/btrfs are a known pathological case for copy-on-write file systems. The database files get horribly fragmented. Btrfs has a (currently disabled per default, because it is not stable) background defragmentation for that, but ZFS does not.
  • Save the UrBackup database on a RAID-5. See http://www.baarf.com/.
  • A lot of full file backups.
  • Enable ZFS deduplication without having enough RAM for the dedup table. See here for a discussion.

Python on Windows issues

Because the file transfer in UrBackup 1.4 was changed, I wanted to test it with a second layer: During the file backup a Python script which goes over all files in all backup paths on the client, selects a random subset and calculates the md5-sums of this random subset. On the server those md5-sums are then verified.

Turns out file tree walking in Python on Windows has a few issues.

First of all, even though the os.walk function has a „followlinks“ parameter, this parameter does not work on Windows with Junctions. In particular it followed the compatibility junctions in the C:\Users\Username folder causing an infinite loop, till the file system complained about the path length (32k). Usually this is not an issue, because users seem to not have the permission to use those junctions, but the UrBackup client backend is running as a SYSTEM-service.

os.path.islink returns False for those junctions.

os.path.realpath does not return the target path of the junction.

To solve this I had to install the Python for Windows extensions and manually call the respective API functions. The Python for Windows extensions does not define the FILE_ATTRIBUTE_REPARSE_POINT file attribute constant. So I had to use the value explicitly. Solution:

def is_reparse_point(path):

    try:
        attrs = win32file.GetFileAttributesW(path)

        return attrs & 1024

    except:
        print("Error with GetFileAttributesW with dir "+path)
        return False

def get_all_files(path):

    output = []

    for root, dirs, files in os.walk(path):
        for name in files:
            output.append(os.path.join(root, name))

        torm = []
        for cdir in dirs:
            if is_reparse_point(os.path.join(root, cdir)):
                torm.append(cdir)

        for cdir in torm:
            dirs.remove(cdir)

    return output

 

Symbolically linking directories during incremental file backups

current_hardlinking_schema

Current hard linking incremental file backup schema

Currently files which are unchanged between incremental file backups are hard linked. This means there exist more than one file paths for the same data. The advantages are

  • The operating systems takes care of managing the data. If there are no file entries pointing at the data it deletes it. Removing a backup means removing a folder.
  • Every file backup looks like a full backup on the backup server. This makes restoring and browsing file backups really easy.

But there are some issues in some cases:

  • On some filesystems and/or slow hard disks the hard link operations are slow (NTFS) and are the bottleneck during backups.
  • Some filesystems only have a limited amount of available file entries (ext e.g.).

With a lot of unchanged files between incremental backups it is almost certain the hard linking operation is the bottleneck.

To improve this situation UrBackup server 1.4 will create symlinks to large directories which are completely unchanged during incremental backups. This avoids a lot of the hard linking operations.

symlinking_schema

Symlinking large unchanged directories to a pooled directory

If, during incremental backups, a large unchanged directory is detected it is moved into a client specific directory pool. Then a symbolic link in the current and last backup to the directory in the pool is created. If the last backup already contains a link to the pool a symlink in the current backup is created to the pooled directory. Because the operating system does not count how many symbolic links point to a directory, the number of symlinks per pooled directory is saved into UrBackups internal database. If a backup is deleted and has symlinks the relevant entries in the database are removed. If there are no entries for the directory in the database the pooled directory is removed.

There are some caveats:

  • There are now symbolic links in the backups. Some tools may not be able to handle them, or need to be told how to handle them.
  • If you rename the backup folder all symbolic links are invalidated and won’t work anymore. The UrBackup server has a tool to fix that, but it will be slow.
  • The reference counting of symbolic links has some tricky corner cases and needs some testing.
  • You cannot just delete file backups anymore as then pooled directories may not be deleted. The UrBackup server includes a tool to detect that, but this might be slow with an increasing amount of directories in the pool.

The symbolic linking of large unchanged directories will be the default behavior in UrBackup server 1.4, though you will be able to deactivate it in the advanced settings. UrBackup server will not symlink directories on btrfs, as there the snapshotting provides a superior method to do fast backups.

Comments should work now

I moved the blog from Sourceforge to a different server, because the spam detection plugins of WordPress do not work on the Sourceforge webspace. The comments should work now. Sorry for the inconvenience.

New Features in UrBackup Server 1.3

It has been a while and there are now a lot new features in UrBackup Server 1.3.

Users of the web interface can download a Client specific installer directly from the server now. The installer has the UrBackup Server information embedded, such that the client automatically connects to the server once it is installed. I’ve also published a script that connects to a UrBackup server, creates a client named like the local computer and then downloads and executes the client installer. This enables a one click setup experience for Internet clients.

The new live log lets you see what the UrBackup server is currently busy with. You can either see all debug level log messages or client specific log messages.

 

 

You can see which files the UrBackup Server is currently working on and the usual log messages which you can also view afterwards via web interface or on the client.

 

A “hidden” feature is now accessible via web interface: You can disable any type of backup for any client.


More fine grained permissions
for the client allow you to prevent the users from starting full file backups, but still allow incremental file backups.

 

The soft client quota allows you to limit the amount of storage each client can use. During the nightly cleanup UrBackup deletes the client’s backups until the storage usage is within the bounds prescribed by the soft client quota. Other than a percentage value you can also use something like “20G” as soft client quota.


You can now have separate backup windows for incremental/full file/image backups.

 


Client-side file hashes
prevent the re-transfer of files that are already on the server, e.g., because another client has the same file. In some situations this drastically reduces the bandwidth requirements and speeds up file backups over Internet.


If you have performance problems with file backups the new file entry cache may help you. If the file entry cache is enabled file entries (a mapping from file hash to file paths) are cached in a separate database which may speed up backups. The caches are automatically created and destroyed if this setting is changed (and the server restarted), but creation may take a long time. LMDB makes heavy use of memory mapped files and is therefore only advisable on a 64bit operating system. It does also create a very large sparse file on Windows. When in doubt use the SQLite cache.

 

 

Backup server for the raspberry pi

Binaries of UrBackup for Raspian are available now here (Update: See the nighlty builds or the “wheezy” folder of Debian of the official releases). Performance isn’t that good on the raspberry pi, obviously. I’d avoid using any kind of compression, for example.

Setup instructions:

First setup the backup storage. I suggest using btrfs. Plug in a USB-hard disk and create a partition, e.g. using cfdisk. Then install btrfs and create the filesystem:

apt-get update
apt-get install btrfs-tools
mkfs.btrfs /dev/sda1
mkdir /media/backup

Add an entry to /etc/fstab to automatically mount the USB-hard disk:

/dev/sda1 /media/backup btrfs defaults,nossd 0 0

Then mount the hard disk:

mount /media/backup

Download the UrBackup server package and install it:

wget http://sourceforge.net/projects/urbackup/files/Server/1.2RC/urbackup-server_1.2-1_armhf.deb/download
dpkg -i urbackup-server_1.2-1_armhf.deb

Resolve the dependencies:

apt-get -f install

Enter e.g. /media/backup/urbackup when it asks for the backup storage path.

UrBackup server should run now and automatically backup clients as soon as you install the UrBackup client on one of your PCs in the local network (from here). If you want to change any settings you should go to the web interface. Let 192.168.0.42 be the IP of your raspberry pi. You can then access the web interface of UrBackup via typing http://192.168.0.42:55414 into your address bar. If you want to access your files you should share /media/backup/urbackup e.g. via samba. There are guides on how to install samba on a raspberry pi (here for example).

Sidenote: I’ve heard that it runs even better on a Cubieboard. It has a SATA-port so the USB-Bottlenet is gone. Unfortunately, they’re even harder to get than Raspberry Pis.

File Backup Storage with Btrfs Snapshots

The Linux kernel 3.6 finally added a feature to btrfs which I needed to easily implement a snapshot file backup storage. This feature will be automatically enabled if the backup storage path points to a btrfs file system and btrfs supports cross-sub-volume reflinks.

A backup with snapshots works like this:

  • Every file backup is put into a separate sub-volume, i.e., a file backup with a name like “121102-1010” is not a normal folder any more. It is a sub-volume in the btrfs file system.
  • Btrfs can make snapshots of sub-volumes. On an incremental backup UrBackup creates a snapshot with the usual name like “121102-2210”. The snapshot has the contents of the last file backup. This snapshot operation is very fast, because no files have to be copied or file entries created. Btrfs does not internally create another copy of the file tree. It only creates copies of parts that we change in the future.
  • UrBackup deletes all the files in the snapshot that have changed or have been deleted since the last backup.
  • Then UrBackup loads the files that have changed or are new. If a file was changed UrBackup only writes changed parts of the file causing btrfs to only save the blocks in the file that are actually different.
  • Once the backup has to be removed UrBackup simply deletes the whole sub-volume.

This has several advantages:

  • Snapshot creation is very fast, causing faster incremental backups.
  • Deleting the file backups is way faster because not every file has to be deleted.
  • Only data that has changed in files between incremental backups is stored in the file system. This drastically reduces the storage requirements, e.g, for large database files.

The cross-device reflinks enable UrBackup to store same files which occur on different sub-volumes only once. On other filesystems this is done using hard links, but those only work on the same file system. Because the Linux kernel sees the sub-volume as a different file system the cross-device reflinks have to be used. There is no disadvantage in using them instead of hard links.

Starting a process for all currently logged in users

For silently updating the client I needed to

  1. Kill all the client GUIs (tray icons) of all users
  2. Stop the background service; Update the files and executables; Start the new background service
  3. Start the client GUI for all currently logged in users

The silent update is done by simply calling the normal NSIS installer with a parameter that makes it non-interactive. Information about how step one is handled is in the previous post. Step two was already in the Installer. Step three remained to be solved.

For that I created a NSIS plugin which checks if the current user is the SYSTEM user. If yes, it starts the executable for all currently logged in users. If not it returns zero. It is used like this:

StrCpy $0 "$INSTDIRUrBackupClient.exe"
startplugin::start
Pop $0
${If} $0 == '0'
Exec "$INSTDIRUrBackupClient.exe"
${EndIf}

Source code is here: http://download.urbackup.org/startplugin.zip
Binary is here (for NSIS Unicode): http://download.urbackup.org/startplugin_bin.zip

Killing 64bit processes from a 32bit NSIS installer

There is no 64bit version of the NSIS (Nullsoft Scriptable Install System). 64Bit processes can only be terminated from 64bit processes, so all the available NSIS plugins, such as the KillProc plugin, do not work. Additionally they do not kill the processes of all users – only those of the currently logged in users – which I wanted as well.

The problem is easily solved by packing an executable into the Installer which is then extracted and called. It is used like this in the UrBackup installer:

${If} ${RunningX64}
File "data_x64KillProc.exe"
nsExec::Exec '"$INSTDIRKillProc.exe" UrBackupClient.exe'
${Else}
File "dataKillProc.exe"
nsExec::Exec '"$INSTDIRKillProc.exe" UrBackupClient.exe'
${EndIf}

The source code of that executable is available here: http://download.urbackup.org/KillProc_src.zip
The binary files here: http://download.urbackup.org/KillProc_bin.zip
It expects only one parameter: The name of the running process to kill.