No BerkeleyDB backend

My previous announcement that there will be a Berkeley DB backend was too hasty. I ran into some (for now) unsolvable problems. I posted them into the official Oracle forums but seem to get no reply there:

The final nail in the coffin was, that the advertised increased concurrency was not present. In my tests it performed even worse then SQLite in WAL journal mode. That and the perceived instability (I had a database corruption once) shine a pretty bad light on Berkeley DB. Maybe the SQL layer for Berkeley DB which I used is not stable yet?

On the plus side the tables I had to denormalize cause a significant speed increase for SQLite as well, so all this work was not for nothing.
Maybe I will revisit Berkeley DB in a few months/years.

MSI installers with next version

I finally bit the bullet and worked on MSI installers for Windows. As anticipated it was not easy. I used WiX.

They do have some advantages over an installer distributed as “exe”:

  • One can add the Microsoft Visual Studio runtime as a “merge module” thus avoiding starting it in the installer manually
  • Apparently installing centralized on domain computers is easier

On the negative side:

  • No shared 32/64 bit MSIs are possible. That means the user has to select the right one before downloading
  • You cannot add custom commands as easily as in NSIS

I think I will only publish 64bit MSIs for now. Most Windows Servers should be 64bit now anyway and I will still publish the “old” installers for users of older and 32bit systems.

Optional BerkeleyDB database backend

In the upcoming version of UrBackup Server you will be able to choose BerkeleyDB as database backend instead of SQLite. I am still deliberating about if I will make it the default in Windows. Not in Linux though as the BerkeleyDB version UrBackup needs is not yet e.g. in Debian stable.

The advantage of BerkeleyDB over SQLite is that it is build for higher concurrency. So if you want to have a lot of simultaneous backups you should definitely use it. The new SQLite compatibility layer of BerkeleyDB also made it very easy to add that alternative backend. (The BerkeleyDB people do not like you calling it a backend. They say it is a SQLite frontend for BerkeleyDB.)
It is not as robust as SQLite though. For example it has some problems when the filesystem the database is saved on is at its capacity limit. In my case it slowed down to a crawl. Also if you do not adjust the database parameters correctly it may throw “out of memory” errors. I am still testing what the correct parameters are. If you just set them really high it needs a lot of memory.
For example it said “Lock table out of lock entries” but increasing the number of lockers such that this error did not occur any more resulted in 1GB more memory usage. This is simply too much. I then tracked the problem down to a table join in which BerkeleyDB seems to need a disproportionately large number of locks. Denormalizing a table, such that this join is no longer necessary solved that problem, I think.
BerkeleyDB may also be slower in situations where no concurrency is involved as it has a much more fine grained locking system and in such situations then locks to much. So using it with only a few clients will cause unnecessary locking overhead.

I plan on automatically converting the database as soon as the BerkeleyDB plugin is loaded into UrBackup Server. So the only inconvenience should be a long database reconstruction time during which the database is converted.
The table denormalization and some index rebuilding will also take place on upgrade. This also took a lot of time on my test system.
Once you switched, converting back will be a manual job. I think I’ll have to write in detail about that once the new version is released.