Linux image backups with UrBackup 2.5.y

So I’ve resisted the idea of Linux image backups for a while. After all, one can just backup all the files, right? True, even with Linuxs fast file access this might take a bit longer than doing an image backup if one has a a lot of files and one wants to back them all up…

Image backups also make backups and restores a lot easier. One doesn’t have to think about what to backup, instead it’s just the whole thing. When restoring, it’s just one step: Copy back the volume.

Advantages:

  • Everything is backed up
  • Content does not affect backup or restore performance (e.g. many small files don’t slow down backups and restores)
  • 1:1 backups and restores of volumes

Disadvantages:

  • Can’t select subset to backup (e.g. exclude temporary file data)
  • Can’t migrate to different file system on restore
  • File level deduplication doesn’t work
  • E.g. defragmentation causes changes to backup

The following are step-by-step instructions on how to backup a Linux VPS.

Backup VPS (Step-by-step)

Assumption: VPS is running Debian 10/buster. 2.5.x server + client is used.

1. Add new client on server web interface:

2. Add active/Internet client

3. Copy & paste Linux client installation command and follow setup:

4. Reboot client/VPS (if not using dattobd)

5. Create image backup (“C” gets automatically translated to the Linux root volume)

Restore VPS (Step-by-step)

1. Browse to image backup in “Backups” tab on web interface

2. Click on restore Linux image

3. Copy & paste into new Debian 10/buster instance and follow instructions:

Ransomware canary

One upcoming feature is the ransomware canary file. The idea is that UrBackup generates a random file that looks like a file that potential ransomware should definitely encrypt. The backup server then checks if this file was changed or deleted during every backup. If ransomware encrypts the file the backup server will notice, fail backups and the backup server admin will get notified via alerts, can easily see the last known good backup and those good backup won’t get replaced by bad backups (backups with ransomware encrypted files).

The setting to configure this is only present on the server. For example it would be set to

Users/*/Documents/^important (don’t delete)/important

Clients would go into the „Users“ backup path, descend into all „Documents“ directories below any folder directly below „Users“. Create a „important (don’t delete)“ folder if it doesn’t exist yet (that’s what the caret does). Then create a (random) „important-[SERVER TOKEN].docx“ canary file in that folder. The owner of the folder and file is set intelligently to the owner of sibling files or parent folders.

For now it is a very basic implementation. I.e.,

  • It only uses a „docx“ file
  • The template „docx“ is bundled with the client and only minimally modified on each client
  • Only works with file backups

If a user moves, deletes or modifies a canary file the remedy to get backups working again would be to set the canary paths to empty for one file backup.

Thoughts? Improvement ideas?

Automatic notarization for macOS

During the effort of building the UrBackup client for macOS the problem of automatically notarizing the build came up. There is only sparse documentation and disparate sources for that. The current build script does this in pure bash now. You can see this and use it as reference here:

https://github.com/uroni/urbackup_backend/blob/1b5d55439aad2a799ca5b55137feb0b72198d0c3/create_osx_installer.sh#L124

Building completely static Linux binaries via Android NDK

The Android NDK can be used to build completely static Linux binaries which run on any Linux.

Advantages:

  • Runs on every Linux distribution be it RHEL, Debian or Alpine
  • Linux (i.e. Linus Torvalds) is pretty strict about backwards compatibility so it’ll continue to run on future Linux kernels without problems
  • Theoretically it’ll even work without a Linux distribution, e.g. in a minimal Docker container
  • All the dependencies are static and confirmed to work together

Disadvantages:

  • Might use more memory at runtime since libraries (such as libc.so) aren’t shared with other programs
  • Behavior might deviate from other programs in the distribution. E.g. when it doesn’t respect /etc/nsswitch.conf, parses /etc/resolv.conf differently or when OpenSSL looks for the root certificates in the wrong place

To mitigate the disadvantages the UrBackup client Linux binary installer first tries to use a glibc (non-static) build on amd64/x86_64 (the most common platform). Only if that doesn’t run (e.g. because the glibc is too old), does it fall back to the NDK build.

Previously this was done with ELLCC, but that doesn’t support C++ exceptions and doesn’t seem to get updated anymore.

Usage

Since UrBackup uses autotools to build, cross compilation is automatically present and can be used simply by setting a few environment variables before building:

export NDK=/path/to/android/ndk/android-ndk-r20
export HOST_TAG=linux-x86_64
export TOOLCHAIN=$NDK/toolchains/llvm/prebuilt/$HOST_TAG
export TARGET=x86_64-linux-android
export TARGET2=${TARGET}29
export AR=$TOOLCHAIN/bin/$TARGET-ar
export AS=$TOOLCHAIN/bin/$TARGET-as              
export CC=$TOOLCHAIN/bin/$TARGET2-clang                
export CXX=$TOOLCHAIN/bin/$TARGET2-clang++                
export LD=$TOOLCHAIN/bin/$TARGET-ld                
export RANLIB=$TOOLCHAIN/bin/$TARGET-ranlib                
export STRIP=$TOOLCHAIN/bin/$TARGET-strip
export NDK_CPUFLAGS=""
 ./configure --enable-headless --enable-c-ares --enable-embedded-cryptopp --enable-embedded-zstd LDFLAGS="-static -Wl,--gc-sections -O2 $NDK_CPUFLAGS -flto" --host $TARGET --with-zlib=$TOOLCHAIN/sysroot/usr --with-crypto-prefix=$TOOLCHAIN/sysroot/usr --with-openssl=$TOOLCHAIN/sysroot/usr CPPFLAGS="-DURB_THREAD_STACKSIZE64=8388608 -DURB_THREAD_STACKSIZE32=1048576 -DURB_WITH_CLIENTUPDATE -ffunction-sections -fdata-sections -ggdb -O2 -flto $ARCH_CPPFLAGS" CFLAGS="-ggdb -O2 -flto $NDK_CPUFLAGS" CXXFLAGS="-ggdb -O2 -flto $NDK_CPUFLAGS -I$NDK/sources/android/cpufeatures/ -DOPENSSL_SEARCH_CA" LIBS="-ldl" 

See also the script that builds the Linux client installer.

The advantage UrBackup has here, is that many dependencies are already bundled with the source code like crypto++, zstd, lua and sqlite. All the dependencies that are not bundled need to be compiled to a static library (for every architecture) and in my case I have put them into $TOOLCHAIN/sysroot/usr/.

Complications

The Android NDK is of course made to build programs for Android. There are significant differences between Android and other Linux distributions. Here is two I found:

If one wants to resolve a DNS name such as example.com to an IP address one usually uses getaddrinfo(). This won’t work with the Android NDK libc (bionic libc), because it uses the Android resolver by calling some Android runtime java code that is obviously not present on non-Android systems. The solution for this problem was to use c-ares instead of the libc to resolve addresses. If you are using a library that resolves addresses that needs to have the option of using c-ares as well (such as cURL).

A call to system() or popen(), calls the shell (usually /bin/sh). The Android libc, however, calls /system/bin/sh instead, which is of course not present on non-Android distributions. The solution was to replace all those calls to an own version. Again, if any library one uses does use those, they’ll need to be replaced.

On x86 and amd64/x86_64 the Android NDK automatically uses SSE4 CPU instructions which older CPUs do not support. Users complained about that and that client should run on as many systems as possible.
To disable SSE4 the Android NDK compiler needs to be passed “-mno-sse4a -mno-sse4.1 -mno-sse4.2 -mno-popcnt” (that was the only way I found). The problem is that some SSE4 instructions are in the libc and libc++. So, the libc and libc++ need to be recompiled with those flags. The bionic libc source code is (unfortunately) NOT part of the Android NDK source code, it is part of the Android source code.
So after downloading 50GB of Android source code for half a day, one needs to change the Go source code of Androids custom build tool (soong). Adjust e.g. build/soong/cc/config/x86_64_device.go, select the correct architecture to build, then fish out the libc.a (and libc++.a) from the output directory and replace the more then dozens of libc.a occurrences in the NDK (no idea which one it actually uses).

One final complication was that crypto++ does actually feature test and then use SSE4 instructions, so they need to be enabled for some crypto++ compile units. If one specifies both “-mno-sse4” and “-msse4”, “-mno-sse4” seems to take precedence. So the solution was to have a compiler wrapper script that removes “-mno-sse4” in such a case.

UrBackup Windows Installer creator

If you wanted to create a generic UrBackup client installer for your internet server you’d have to create one yourself following instructions here .

I have made https://installercreator.urbackup.org/ to make this process easier. Given the correct parameters it’ll create a executable for you that’ll create a client on your server, then download the client installer for this newly created client and then run this installer. It’ll use the computer name as default client name, optionally appending a random string, such that name collisions do not occur.

Skewed Color

In which I debug why the tray icon on my laptop doesn’t turn yellow during backups anymore and just cannot find the issue:

Turns out I have f.lux on and it is night time and f.lux makes the yellow look like white. Now is that a bug or not?

How I backup LVM volumes on my Xen server

I’ve got a Xen server which runs a couple of Linux and Windows VMs. The VMs are stored in LVM volumes on a LVM volume group which is on a bcache device. The bcache device consists of a mirrored SSD pair (using mdraid) as cache and a mirrored HDD pair (also using mdraid) as backing storage. The SSD caching gives a nice performance boost, but nowadays I would go with SSD storage only, because bcache caused some problems (did not play nice with udev during boot).
The Windows VMs are backed up by installing the UrBackup client in the VMs. To restore I’d need to boot the restore CD in Xen or restore the Windows images via command line in the hypervisor.
The Linux VMs are backed up at hypervisor level in the Xen dom0 (which is Debian in this case) using LVM snapshots. To create and remove LVM snapshots I have following snapshot creation and removal script (the volume group on which the volumes are is mirror-vg).

Snapshot creation script at /usr/local/etc/urbackup/create_filesystem_snapshot:

#!/bin/bash
set -e
SNAP_UID=$1
VOLNAME="$5"
VGNAME="mirror-vg"
if [[ $VOLNAME == "" ]]; then
        echo "No volume name specified"
        exit 1
fi
if [[ $VOLNAME == "other-data" ]]; then
        VGNAME="data2-vg"
fi
if [[ $SNAP_UID == "" ]]; then
        echo "No snapshot uid specified"
        exit 1
fi
export LVM_SUPPRESS_FD_WARNINGS=1
lvcreate -l100%FREE -s -n $SNAP_UID /dev/$VGNAME/$VOLNAME
SUCCESS=0
trap 'test $SUCCESS = 1 || lvremove -f /dev/$VGNAME/$SNAP_UID' EXIT
mkdir -p /mnt/urbackup_snaps/${SNAP_UID}
mount -o ro /dev/$VGNAME/$SNAP_UID /mnt/urbackup_snaps/${SNAP_UID}
SUCCESS=1
echo "SNAPSHOT=/mnt/urbackup_snaps/$SNAP_UID"
exit 0

Snapshot removal script at /usr/local/etc/urbackup/remove_filesystem_snapshot:

#!/bin/bash
set -e
SNAP_UID=$1
SNAP_MOUNTPOINT="$2"
if [[ $SNAP_UID == "" ]]; then
        echo "No snapshot uid specified"
        exit 1
fi
if [[ "$SNAP_MOUNTPOINT" == "" ]]; then
        echo "Snapshot mountpoint is empty"
        exit 1
fi
if ! test -e $SNAP_MOUNTPOINT; then
        echo "Snapshot at $SNAP_MOUNTPOINT was already removed"
        exit 0
fi
if ! df -T -P | egrep "${SNAP_MOUNTPOINT}\$" > /dev/null 2>&1; then
        echo "Snapshot is not mounted. Already removed"
        rmdir "${SNAP_MOUNTPOINT}"
        exit 0
fi
if lsblk -r --output "NAME,MOUNTPOINT" --paths > /dev/null 2>&1; then
        VOLNAME=`lsblk -r --output "NAME,MOUNTPOINT" --paths | egrep " ${SNAP_MOUNTPOINT}\$" | head -n 1 | tr -s " " | cut -d" " -f1`
else
        VOLNAME=`lsblk -r --output "NAME,MOUNTPOINT" | egrep " ${SNAP_MOUNTPOINT}\$" | head -n 1 | tr -s " " | cut -d" " -f1`
        VOLNAME="/dev/mapper/$VOLNAME"
fi
if [ "x$VOLNAME" = x ]; then
    echo "Could not find LVM volume for mountpoint ${SNAP_MOUNTPOINT}"
    exit 1
fi
if [ ! -e "$VOLNAME" ]; then
    echo "LVM volume for mountpoint ${SNAP_MOUNTPOINT} does not exist"
    exit 1
fi
echo "Unmounting $VOLNAME at /mnt/urbackup_snaps/${SNAP_UID}..."
if ! umount /mnt/urbackup_snaps/${SNAP_UID}; then
        sleep 10
        umount /mnt/urbackup_snaps/${SNAP_UID}
fi
rmdir "${SNAP_MOUNTPOINT}"
echo "Destroying LVM snapshot $VOLNAME..."
export LVM_SUPPRESS_FD_WARNINGS=1
lvremove -f "$VOLNAME"

The snapshot scripts are specified via the file /usr/local/etc/urbackup/snapshot.cfg:

create_filesystem_snapshot=/usr/local/etc/urbackup/create_filesystem_snapshot
remove_filesystem_snapshot=/usr/local/etc/urbackup/remove_filesystem_snapshot
volumes_mounted_locally=0

Then I have a virtual client for each LVM volume that needs to be backed up. I have put those virtual clients in a settings group with the default path to backup “/|root/require_snapshot”.

For restore I need to recreate the LVM volume. Create a file system on it (e.g. with mkfs.ext4) mount it in the hypervisor and then restore via.

urbackupclientctl restore-start --virtual-client VOLUMENAME -b last –map-from / --map-to /mnt/localmountpoint

Windows Backup API support in UrBackup 2.1.x

UrBackup 2.1.x has more completselect_windows_componentse Windows Backup API support. Previously the backup API was only used to create snapshots of the specified paths to backup (volume shadow copy snapshots). Now it does a so called component level backup, if configured to do so. You can select the components to backup via the client user interface and then the selected components automatically communicate to UrBackup which files need to be backed up, and on restore it communicates where the files should be restored and if e.g. services should be restarted during/after restore.select_restore_components

This works with applications which in turn support the Windows Backup API, such as for example Microsoft Exchange, Microsoft SQL Server, Microsoft Hyper-V, Oracle DB on Windows.

Testing backup and resrestore_componentstore with those different applications is now the big item on the to-do list. Every help and pointers to applications where backup or restore is broken will be helpful.

Visual Studio 2015 runtime and MSI installer

If you are using the MSI installer to install either UrBackup Client or Server on Windows there is a potential problem you might run into.

Starting with Visual Studio 2015, with which UrBackup Client/Server 2.x are compiled, Microsoft decided to split the Visual Studio runtime into a operating system level component (Universal Runtime) and another “normal” runtime component, wheras earlier it was only a “normal” runtime component.
The operating system component is installed via Windows Update and cannot be installed by a MSI installer. With Windows 10 it is always installed, but with Windows Vista or 8.1 the system needs to be up to date in order for the system component to be present (KB2999226), otherwise UrBackup will not start.

Another work-around is to use the .EXE (NSIS) Installer which includes the operating system compontent. Installation depends on Windows Update functioning correctly (which it may not).

Let’s just say this change does not make the software developer’s and user’s life easier.

New in UrBackup 2.0.x

Wactivitieseb interface modernization. The web interface was a little bit utilitarian which gave many people the wrong impression. With the help of mombojuice the web interface was improved such that it looks much more modern and professional. Many small improvements were made as well. For example the dates are now formatted according to browser locale, backups can be started via drop-down menu and the live log of a running backup can be directly accessed from the activities screen.

Improved file deduplication. Completely reworked the file deduplication and file backup statistics calculation. This should be much faster, scalable and reliable now.

File lastmodifiedbackup improvements. File meta-data such as last modified time and file permissions are now backed up on all supported client systems (Windows, Linux, Mac OS X). Supporting more exotic file system features such as sparse files UrBackup is now a fully featured file backup solution.

File backup restore. To restore the file meta-data UrBackup has now an integrated file restore. The file restore reuses client-side hashes, if present, and transfers only differences, such that restoring folders with only few changes since the restored backup is fast.

access_backupsDirect backup access. If configured, the backed up file permissions are used to allow clients direct access to their files with only minimal configuration. On Windows there is a shortcut in Explorer which directly opens the relevant/file folder in the browser. There is a new list view which shows a file/folder in all backups. For files, hashes are used to show when the file content changed (versions).

image_backup_settingsImage backup improvements. UrBackup supports GPT formatted disks now and the restore CD boots on UEFI firmware devices (also with secure boot enabled). In combination with btrfs, UrBackup supports an incremental forever style image backup and image backups over 2TB. For VHD/VHDZ UrBackup has now settings to base incremental backups on the last or last full image backup. Full image backups can be configured to be synthetic full backups transferring only changes since the last image backup.

Significant security improvements. Forward secrecy for Internet clients via ECDH and Internet client security improvement by using AES-GCM. Switch from DSA to ECDSA for client update and server identity signatures. Web server/restore CD login now uses PBKDF2.

Mac OS X client. There is nowmac_backup_running a UrBackup Client for Mac OS X. This client is fully featured, excluding image backup (like Linux client). The Mac OS X client can be used as a technically superior backup solution to Time Machine.

Improved command line. Mainlttyy for Linux all command line usage has been significantly improved. This includes the command line client (urbackupclientctl), the server command line (urbackupsrv) and the restore client.

Linux file system snapshotting. Snapshotting now also works on Linux and is fully integrated. A portable Linux client includes snapshot scripts for LVM, dattobd and btrfs which work without changes in most cases.

Lots of other changes. Proper symbolic link backup. Virtual clients allow you to backup different sets of files at different intervals and max/min amounts. Simultaneous image and file backups. Different backup speeds and backup intervals at different times. Improved Internet transfer compression. New hashing method where the server only needs to hash changed parts of a file.