The Android NDK can be used to build completely static Linux binaries which run on any Linux.

Advantages:

• Runs on every Linux distribution be it RHEL, Debian or Alpine
• Linux (i.e. Linus Torvalds) is pretty strict about backwards compatibility so it’ll continue to run on future Linux kernels without problems
• Theoretically it’ll even work without a Linux distribution, e.g. in a minimal Docker container
• All the dependencies are static and confirmed to work together

Disadvantages:

• Might use more memory at runtime since libraries (such as libc.so) aren’t shared with other programs
• Behavior might deviate from other programs in the distribution. E.g. when it doesn’t respect /etc/nsswitch.conf, parses /etc/resolv.conf differently or when OpenSSL looks for the root certificates in the wrong place

To mitigate the disadvantages the UrBackup client Linux binary installer first tries to use a glibc (non-static) build on amd64/x86_64 (the most common platform). Only if that doesn’t run (e.g. because the glibc is too old), does it fall back to the NDK build.

Previously this was done with ELLCC, but that doesn’t support C++ exceptions and doesn’t seem to get updated anymore.

Usage

Since UrBackup uses autotools to build, cross compilation is automatically present and can be used simply by setting a few environment variables before building:

export NDK=/path/to/android/ndk/android-ndk-r20
export HOST_TAG=linux-x86_64
export TOOLCHAIN=$NDK/toolchains/llvm/prebuilt/$HOST_TAG
export TARGET=x86_64-linux-android
export TARGET2=${TARGET}29
export AR=$TOOLCHAIN/bin/$TARGET-ar
export AS=$TOOLCHAIN/bin/$TARGET-as              
export CC=$TOOLCHAIN/bin/$TARGET2-clang                
export CXX=$TOOLCHAIN/bin/$TARGET2-clang++                
export LD=$TOOLCHAIN/bin/$TARGET-ld                
export RANLIB=$TOOLCHAIN/bin/$TARGET-ranlib                
export STRIP=$TOOLCHAIN/bin/$TARGET-strip
export NDK_CPUFLAGS=""
 ./configure --enable-headless --enable-c-ares --enable-embedded-cryptopp --enable-embedded-zstd LDFLAGS="-static -Wl,--gc-sections -O2 $NDK_CPUFLAGS -flto" --host $TARGET --with-zlib=$TOOLCHAIN/sysroot/usr --with-crypto-prefix=$TOOLCHAIN/sysroot/usr --with-openssl=$TOOLCHAIN/sysroot/usr CPPFLAGS="-DURB_THREAD_STACKSIZE64=8388608 -DURB_THREAD_STACKSIZE32=1048576 -DURB_WITH_CLIENTUPDATE -ffunction-sections -fdata-sections -ggdb -O2 -flto $ARCH_CPPFLAGS" CFLAGS="-ggdb -O2 -flto $NDK_CPUFLAGS" CXXFLAGS="-ggdb -O2 -flto $NDK_CPUFLAGS -I$NDK/sources/android/cpufeatures/ -DOPENSSL_SEARCH_CA" LIBS="-ldl" 

See also the script that builds the Linux client installer.

The advantage UrBackup has here, is that many dependencies are already bundled with the source code like crypto++, zstd, lua and sqlite. All the dependencies that are not bundled need to be compiled to a static library (for every architecture) and in my case I have put them into $TOOLCHAIN/sysroot/usr/.

Complications

The Android NDK is of course made to build programs for Android. There are significant differences between Android and other Linux distributions. Here is two I found:

If one wants to resolve a DNS name such as example.com to an IP address one usually uses getaddrinfo(). This won’t work with the Android NDK libc (bionic libc), because it uses the Android resolver by calling some Android runtime java code that is obviously not present on non-Android systems. The solution for this problem was to use c-ares instead of the libc to resolve addresses. If you are using a library that resolves addresses that needs to have the option of using c-ares as well (such as cURL).

A call to system() or popen(), calls the shell (usually /bin/sh). The Android libc, however, calls /system/bin/sh instead, which is of course not present on non-Android distributions. The solution was to replace all those calls to an own version. Again, if any library one uses does use those, they’ll need to be replaced.

On x86 and amd64/x86_64 the Android NDK automatically uses SSE4 CPU instructions which older CPUs do not support. Users complained about that and that client should run on as many systems as possible.
To disable SSE4 the Android NDK compiler needs to be passed “-mno-sse4a -mno-sse4.1 -mno-sse4.2 -mno-popcnt” (that was the only way I found). The problem is that some SSE4 instructions are in the libc and libc++. So, the libc and libc++ need to be recompiled with those flags. The bionic libc source code is (unfortunately) NOT part of the Android NDK source code, it is part of the Android source code.
So after downloading 50GB of Android source code for half a day, one needs to change the Go source code of Androids custom build tool (soong). Adjust e.g. build/soong/cc/config/x86_64_device.go, select the correct architecture to build, then fish out the libc.a (and libc++.a) from the output directory and replace the more then dozens of libc.a occurrences in the NDK (no idea which one it actually uses).

One final complication was that crypto++ does actually feature test and then use SSE4 instructions, so they need to be enabled for some crypto++ compile units. If one specifies both “-mno-sse4” and “-msse4”, “-mno-sse4” seems to take precedence. So the solution was to have a compiler wrapper script that removes “-mno-sse4” in such a case.