Raspberry PI 4 Backup Appliance update

An update for the Raspberry PI 4 backup appliance (last post). Raspberry PI specific changes:

  • Upgrade Linux kernel to 5.4.y. This allows e.g. a swap file on the system disk
  • Removed the USB 3.0 UAS driver, since it doesn’t seem to work (with the disks I tested it with — see also this thread). This makes USB disks slower, but at least it works reliably
  • Use xchacha12+adiantum as system disk encryption method since RPI4 doesn’t have AES CPU instructions. This significantly increases performance

    If you want to use it manually outside the appliance use
    cryptsetup luksFormat --cipher xchacha12,aes-adiantum-plain64 --type luks2 --sector-size 4096
    to format the encrypted disk

Download the image for the SD card.

Write the image to the SD card using e.g. Win32 Disk Imager.

Discussion

Discuss the Infscape UrBackup Appliance here.

Raspberry Pi 4 backup appliance

Rapsberry PI 4 is (finally) relatively powerful hardware and suited to host a UrBackup appliance. Specifically:

  • 4 GB RAM
  • 64 bit support
  • USB 3
  • Gigabit ethernet
  • Build-in Wifi

The RAM is of course not ECC-RAM which makes it unsuited for large UrBackup appliance instances or serious use (you could always risk it, see notes on ECC RAM section on the bottom).

The main advantages of Raspberry Pis are that they are widely available, have good support and merely by having so many people focused on them, having few bugs.

A few gotchas when setting up a UrBackup server:

  • Use the SD card as a read-only device for booting the operating system. Regularly writing to the SD card will break it because SD cards are not designed for such use. There are “industrial” SD cards designed for regular (random) writes (e.g. SwissBit) but they cost way too much (an external USB SSD is cheaper)
  • I had a lot of problems getting the 64bit Linux kernel to boot, but I guess this will improve
  • Power supply is still an issue. The USB SSD drive I was using caused issues because (I think) the power supplied by the Raspberry wasn’t enough. I’d guess the Raspberry + USB SDD need a 4A power supply at least.

The system plus costs (w.o. taxes):

Raspberry PI 4 Model B 4GB50 €
Intenso SSD 256GB37 €
32GB microSD SDHC3 €
Raspberry Pi 4 Case6 €
Powered USB 3 hub 5V/4A (20W)20 €
USB 3 to USB C cable7 €
Sum (w.o. taxes)123 €

Once you have the base system, you can add USB disks as backup storage, use those 2-bay docking stations + disks or use larger 4-bay USB external disk enclosures. Adding single USB disks is probably cheapest.
Alternatively, if using the Infscape UrBackup Appliance, you can store your backups directly to the cloud (Amazon S3 or compatible) using the SSD as local cache.

Software

I’ll describe how to setup the Infscape UrBackup Appliance. You can of course also install Raspbian and install UrBackup on that.

Download the image for the SD card.

Write the image to the SD card using e.g. Win32 Disk Imager.

For setup, you’ll have to attach the Raspberry PI to your wired ethernet network. After setup, you can run it via Wifi only. Put the SD card into the Raspberry and start it up. After boot it’ll show its IP address on its display. Alternatively, you can look for new devices on your router, or using an IP range scanning tool.

Browse to http://YOURRASPBERRYIP and follow the setup wizard.

Make sure the SSD is attached and click on “+Use device” on the status screen. Then use the SSD as external system disk (UrBackup database, logs and temporary files will be stored on the SSD instead of the SD card).

If you want to store the backups to S3, select that you want to use the system disk as cache (or attach an additional SSD as cache only disk). If you want to store the backups to a local (USB) disks I’d recommend to also use the system disk as cache for the local USB disks (cached, auto-layout RAID).

ECC-RAM

If you are unlucky a single bit error in your Raspberry PI RAM can make the whole backup storage unwritable. In that case you’ll have to either make a new backup server and replicate everything (that is still readable) to that one, or simply start with a new backup server. The worst case is that it corrupts data in such a way that it does not detect the corruption, in which case you’ll only find there is a problem when you restore/actually want to use the backup.
That said, the probability of the RAM having problems is perhaps very low. We can probably trust the Raspberry PI foundation to have selected reliable RAM plus settings that minimize the risk of corruption.
The cloud/RAID backup storage has an advanced option “Size of in-memory write-back cache”. Per default it is disabled and you probably shouldn’t use it if you don’t have ECC-RAM as it increases the time data is kept in memory before it is written to checksum protected storage.

Discussion

Discuss the Infscape UrBackup Appliance here.

Connect clients with a HTTPS CONNECT web proxy

With 2.4.x you can use UrBackup with a HTTPS proxy. This way you can have the web interface and the clients connecting at the same port, secured by the same transport encryption (SSL). This post shows how to do this in combination with the Apache web server.

The idea is that the client connects to the web server and issues a HTTP CONNECT request to the actual UrBackup server.

First Enable CONNECT proxy module in apache. On debian via

a2enmod proxy_connect  

Then allow connections to the UrBackup server Internet port by adding

AllowConnect 55415

to your apache configuration.

Next in your apache virtual host configuration, set proxy options such as the timeout, allow proxy connections to the UrBackup server, and disallow them to every other host:

ProxyTimeout 600
ProxyRequests On  
<Proxy 127.0.0.1:55415>
</Proxy>
<ProxyMatch ^(?!127.0.0.1:55415$).*$>
    Order Deny,Allow
    Deny from all
</ProxyMatch>

Then, go to your UrBackup server web interface and setup your web server URL as Internet client proxy (https://example.com) and the Internet server name/IP as 127.0.0.1. Internet clients should then start connecting via your web server to your UrBackup server. Once all clients connect this way you could turn off UrBackup’s build in Internet transfer encryption and rely on SSL.

Fixing client IP addresses

You may notice that on the status page all Internet clients now show the IP address of your web server as their IP address. Fixing this is a bit difficult, as there is no standard way to forward the client IP address information from the web server (compared to a normal HTTP proxy where there is a X-Forwarded-For header). So, a bit of hacking to fix this is in order. I modified the mod_proxy_connect apache plugin to forward the client IP information in a 50 byte buffer to the backend: mod_proxy_connect.c
On debian you could replace your original mod_proxy_connect with the modified one via the following commands:

apt install apache2-dev
wget https://gist.githubusercontent.com/uroni/143c0d7ed6169e89f2d6c59a870dd4cc/raw/28dd30b1f82938777c504f2afdc5f162fd91b3fd/mod_proxy_connect.c
apxs -i -a -c mod_proxy_connect.c

Then in the UrBackup server advanced global settings set “List of server IPs (proxys) from which to expect endpoint information (forwarded for) when connecting to Internet service (needs server restart)” to include your web server IP (127.0.0.1 in the example here). After a server restart you should be able to see the actual client IP instead of the web server IP on the status screen.

Fixing SNI errors

If you have multiple virtual hosts with SSL there is an issue with SNI. Apache2 automatically compares the hostname in the CONNECT request with the server name in the SSL connection (SNI) and rejects the request if they differ. The only solution (or ugly hack) I found to fix this was to add the hostname with the target IP to /etc/hosts and then use the hostname instead of the IP in the CONNECT request. I.e., add “127.0.0.1 example.com” to /etc/hosts, then replace 127.0.0.1 with example.com in all the configuration above.

Additional proxy authentication

As additional security layer, one can require proxy authentication. Clients need to know a username+password to get through the web server to the UrBackup server. With apache2 e.g.:

 htpasswd -c -b /etc/apache2/urbackup_password urbackup passw0rd 

Then modify the proxy section to:

<Proxy 127.0.0.1:55415>
    AuthType Basic
    AuthName "Restricted UrBackup"
    AuthBasicProvider file
    AuthUserFile "/etc/apache2/urbackup_password"
    Require user urbackup
</Proxy>

Afterward add username+password to the proxy url, that is e.g. https://urbackup:password@example.com

UrBackup Windows Installer creator

If you wanted to create a generic UrBackup client installer for your internet server you’d have to create one yourself following instructions here .

I have made https://installercreator.urbackup.org/ to make this process easier. Given the correct parameters it’ll create a executable for you that’ll create a client on your server, then download the client installer for this newly created client and then run this installer. It’ll use the computer name as default client name, optionally appending a random string, such that name collisions do not occur.

Image mounting in UrBackup Server 2.1.x

UrBackup 2.1.x can now mount image backups. That is, it lists the image backups it has on the web interface, you can browse into them and e.g. download files or directories as a ZIP file.
I am particularly proud that this works on Linux as well as on Windows with both raw image files and VHD(z) files. On FreeBSD it only works with raw image files currently. The screenshots are from a Windows UrBackup server.

On Linux it uses libguestfs-tools to mount images in a sandboxed virtual machine. On Windows/FreeBSD mounting a hostile image may be a dangerous operation.

How I backup LVM volumes on my Xen server

I’ve got a Xen server which runs a couple of Linux and Windows VMs. The VMs are stored in LVM volumes on a LVM volume group which is on a bcache device. The bcache device consists of a mirrored SSD pair (using mdraid) as cache and a mirrored HDD pair (also using mdraid) as backing storage. The SSD caching gives a nice performance boost, but nowadays I would go with SSD storage only, because bcache caused some problems (did not play nice with udev during boot).
The Windows VMs are backed up by installing the UrBackup client in the VMs. To restore I’d need to boot the restore CD in Xen or restore the Windows images via command line in the hypervisor.
The Linux VMs are backed up at hypervisor level in the Xen dom0 (which is Debian in this case) using LVM snapshots. To create and remove LVM snapshots I have following snapshot creation and removal script (the volume group on which the volumes are is mirror-vg).

Snapshot creation script at /usr/local/etc/urbackup/create_filesystem_snapshot:

#!/bin/bash
set -e
SNAP_UID=$1
VOLNAME="$5"
VGNAME="mirror-vg"
if [[ $VOLNAME == "" ]]; then
        echo "No volume name specified"
        exit 1
fi
if [[ $VOLNAME == "other-data" ]]; then
        VGNAME="data2-vg"
fi
if [[ $SNAP_UID == "" ]]; then
        echo "No snapshot uid specified"
        exit 1
fi
export LVM_SUPPRESS_FD_WARNINGS=1
lvcreate -l100%FREE -s -n $SNAP_UID /dev/$VGNAME/$VOLNAME
SUCCESS=0
trap 'test $SUCCESS = 1 || lvremove -f /dev/$VGNAME/$SNAP_UID' EXIT
mkdir -p /mnt/urbackup_snaps/${SNAP_UID}
mount -o ro /dev/$VGNAME/$SNAP_UID /mnt/urbackup_snaps/${SNAP_UID}
SUCCESS=1
echo "SNAPSHOT=/mnt/urbackup_snaps/$SNAP_UID"
exit 0

Snapshot removal script at /usr/local/etc/urbackup/remove_filesystem_snapshot:

#!/bin/bash
set -e
SNAP_UID=$1
SNAP_MOUNTPOINT="$2"
if [[ $SNAP_UID == "" ]]; then
        echo "No snapshot uid specified"
        exit 1
fi
if [[ "$SNAP_MOUNTPOINT" == "" ]]; then
        echo "Snapshot mountpoint is empty"
        exit 1
fi
if ! test -e $SNAP_MOUNTPOINT; then
        echo "Snapshot at $SNAP_MOUNTPOINT was already removed"
        exit 0
fi
if ! df -T -P | egrep "${SNAP_MOUNTPOINT}\$" > /dev/null 2>&1; then
        echo "Snapshot is not mounted. Already removed"
        rmdir "${SNAP_MOUNTPOINT}"
        exit 0
fi
if lsblk -r --output "NAME,MOUNTPOINT" --paths > /dev/null 2>&1; then
        VOLNAME=`lsblk -r --output "NAME,MOUNTPOINT" --paths | egrep " ${SNAP_MOUNTPOINT}\$" | head -n 1 | tr -s " " | cut -d" " -f1`
else
        VOLNAME=`lsblk -r --output "NAME,MOUNTPOINT" | egrep " ${SNAP_MOUNTPOINT}\$" | head -n 1 | tr -s " " | cut -d" " -f1`
        VOLNAME="/dev/mapper/$VOLNAME"
fi
if [ "x$VOLNAME" = x ]; then
    echo "Could not find LVM volume for mountpoint ${SNAP_MOUNTPOINT}"
    exit 1
fi
if [ ! -e "$VOLNAME" ]; then
    echo "LVM volume for mountpoint ${SNAP_MOUNTPOINT} does not exist"
    exit 1
fi
echo "Unmounting $VOLNAME at /mnt/urbackup_snaps/${SNAP_UID}..."
if ! umount /mnt/urbackup_snaps/${SNAP_UID}; then
        sleep 10
        umount /mnt/urbackup_snaps/${SNAP_UID}
fi
rmdir "${SNAP_MOUNTPOINT}"
echo "Destroying LVM snapshot $VOLNAME..."
export LVM_SUPPRESS_FD_WARNINGS=1
lvremove -f "$VOLNAME"

The snapshot scripts are specified via the file /usr/local/etc/urbackup/snapshot.cfg:

create_filesystem_snapshot=/usr/local/etc/urbackup/create_filesystem_snapshot
remove_filesystem_snapshot=/usr/local/etc/urbackup/remove_filesystem_snapshot
volumes_mounted_locally=0

Then I have a virtual client for each LVM volume that needs to be backed up. I have put those virtual clients in a settings group with the default path to backup “/|root/require_snapshot”.

For restore I need to recreate the LVM volume. Create a file system on it (e.g. with mkfs.ext4) mount it in the hypervisor and then restore via.

urbackupclientctl restore-start --virtual-client VOLUMENAME -b last –map-from / --map-to /mnt/localmountpoint

Windows Backup API support in UrBackup 2.1.x

UrBackup 2.1.x has more completselect_windows_componentse Windows Backup API support. Previously the backup API was only used to create snapshots of the specified paths to backup (volume shadow copy snapshots). Now it does a so called component level backup, if configured to do so. You can select the components to backup via the client user interface and then the selected components automatically communicate to UrBackup which files need to be backed up, and on restore it communicates where the files should be restored and if e.g. services should be restarted during/after restore.select_restore_components

This works with applications which in turn support the Windows Backup API, such as for example Microsoft Exchange, Microsoft SQL Server, Microsoft Hyper-V, Oracle DB on Windows.

Testing backup and resrestore_componentstore with those different applications is now the big item on the to-do list. Every help and pointers to applications where backup or restore is broken will be helpful.

Visual Studio 2015 runtime and MSI installer

If you are using the MSI installer to install either UrBackup Client or Server on Windows there is a potential problem you might run into.

Starting with Visual Studio 2015, with which UrBackup Client/Server 2.x are compiled, Microsoft decided to split the Visual Studio runtime into a operating system level component (Universal Runtime) and another “normal” runtime component, wheras earlier it was only a “normal” runtime component.
The operating system component is installed via Windows Update and cannot be installed by a MSI installer. With Windows 10 it is always installed, but with Windows Vista or 8.1 the system needs to be up to date in order for the system component to be present (KB2999226), otherwise UrBackup will not start.

Another work-around is to use the .EXE (NSIS) Installer which includes the operating system compontent. Installation depends on Windows Update functioning correctly (which it may not).

Let’s just say this change does not make the software developer’s and user’s life easier.

New in UrBackup 2.0.x

Wactivitieseb interface modernization. The web interface was a little bit utilitarian which gave many people the wrong impression. With the help of mombojuice the web interface was improved such that it looks much more modern and professional. Many small improvements were made as well. For example the dates are now formatted according to browser locale, backups can be started via drop-down menu and the live log of a running backup can be directly accessed from the activities screen.

Improved file deduplication. Completely reworked the file deduplication and file backup statistics calculation. This should be much faster, scalable and reliable now.

File lastmodifiedbackup improvements. File meta-data such as last modified time and file permissions are now backed up on all supported client systems (Windows, Linux, Mac OS X). Supporting more exotic file system features such as sparse files UrBackup is now a fully featured file backup solution.

File backup restore. To restore the file meta-data UrBackup has now an integrated file restore. The file restore reuses client-side hashes, if present, and transfers only differences, such that restoring folders with only few changes since the restored backup is fast.

access_backupsDirect backup access. If configured, the backed up file permissions are used to allow clients direct access to their files with only minimal configuration. On Windows there is a shortcut in Explorer which directly opens the relevant/file folder in the browser. There is a new list view which shows a file/folder in all backups. For files, hashes are used to show when the file content changed (versions).

image_backup_settingsImage backup improvements. UrBackup supports GPT formatted disks now and the restore CD boots on UEFI firmware devices (also with secure boot enabled). In combination with btrfs, UrBackup supports an incremental forever style image backup and image backups over 2TB. For VHD/VHDZ UrBackup has now settings to base incremental backups on the last or last full image backup. Full image backups can be configured to be synthetic full backups transferring only changes since the last image backup.

Significant security improvements. Forward secrecy for Internet clients via ECDH and Internet client security improvement by using AES-GCM. Switch from DSA to ECDSA for client update and server identity signatures. Web server/restore CD login now uses PBKDF2.

Mac OS X client. There is nowmac_backup_running a UrBackup Client for Mac OS X. This client is fully featured, excluding image backup (like Linux client). The Mac OS X client can be used as a technically superior backup solution to Time Machine.

Improved command line. Mainlttyy for Linux all command line usage has been significantly improved. This includes the command line client (urbackupclientctl), the server command line (urbackupsrv) and the restore client.

Linux file system snapshotting. Snapshotting now also works on Linux and is fully integrated. A portable Linux client includes snapshot scripts for LVM, dattobd and btrfs which work without changes in most cases.

Lots of other changes. Proper symbolic link backup. Virtual clients allow you to backup different sets of files at different intervals and max/min amounts. Simultaneous image and file backups. Different backup speeds and backup intervals at different times. Improved Internet transfer compression. New hashing method where the server only needs to hash changed parts of a file.