Optional BerkeleyDB database backend

In the upcoming version of UrBackup Server you will be able to choose BerkeleyDB as database backend instead of SQLite. I am still deliberating about if I will make it the default in Windows. Not in Linux though as the BerkeleyDB version UrBackup needs is not yet e.g. in Debian stable.

The advantage of BerkeleyDB over SQLite is that it is build for higher concurrency. So if you want to have a lot of simultaneous backups you should definitely use it. The new SQLite compatibility layer of BerkeleyDB also made it very easy to add that alternative backend. (The BerkeleyDB people do not like you calling it a backend. They say it is a SQLite frontend for BerkeleyDB.)
It is not as robust as SQLite though. For example it has some problems when the filesystem the database is saved on is at its capacity limit. In my case it slowed down to a crawl. Also if you do not adjust the database parameters correctly it may throw “out of memory” errors. I am still testing what the correct parameters are. If you just set them really high it needs a lot of memory.
For example it said “Lock table out of lock entries” but increasing the number of lockers such that this error did not occur any more resulted in 1GB more memory usage. This is simply too much. I then tracked the problem down to a table join in which BerkeleyDB seems to need a disproportionately large number of locks. Denormalizing a table, such that this join is no longer necessary solved that problem, I think.
BerkeleyDB may also be slower in situations where no concurrency is involved as it has a much more fine grained locking system and in such situations then locks to much. So using it with only a few clients will cause unnecessary locking overhead.

I plan on automatically converting the database as soon as the BerkeleyDB plugin is loaded into UrBackup Server. So the only inconvenience should be a long database reconstruction time during which the database is converted.
The table denormalization and some index rebuilding will also take place on upgrade. This also took a lot of time on my test system.
Once you switched, converting back will be a manual job. I think I’ll have to write in detail about that once the new version is released.

Leave a Reply

Your email address will not be published. Required fields are marked *