In the upcoming version of UrBackup Server you will be able to choose BerkeleyDB as database backend instead of SQLite. I am still deliberating about if I will make it the default in Windows. Not in Linux though as the BerkeleyDB version UrBackup needs is not yet e.g. in Debian stable.

The advantage of BerkeleyDB over SQLite is that it is build for higher concurrency. So if you want to have a lot of simultaneous backups you should definitely use it. The new SQLite compatibility layer of BerkeleyDB also made it very easy to add that alternative backend. (The BerkeleyDB people do not like you calling it a backend. They say it is a SQLite frontend for BerkeleyDB.)
It is not as robust as SQLite though. For example it has some problems when the filesystem the database is saved on is at its capacity limit. In my case it slowed down to a crawl. Also if you do not adjust the database parameters correctly it may throw “out of memory” errors. I am still testing what the correct parameters are. If you just set them really high it needs a lot of memory.
For example it said “Lock table out of lock entries” but increasing the number of lockers such that this error did not occur any more resulted in 1GB more memory usage. This is simply too much. I then tracked the problem down to a table join in which BerkeleyDB seems to need a disproportionately large number of locks. Denormalizing a table, such that this join is no longer necessary solved that problem, I think.
BerkeleyDB may also be slower in situations where no concurrency is involved as it has a much more fine grained locking system and in such situations then locks to much. So using it with only a few clients will cause unnecessary locking overhead.

I plan on automatically converting the database as soon as the BerkeleyDB plugin is loaded into UrBackup Server. So the only inconvenience should be a long database reconstruction time during which the database is converted.
The table denormalization and some index rebuilding will also take place on upgrade. This also took a lot of time on my test system.
Once you switched, converting back will be a manual job. I think I’ll have to write in detail about that once the new version is released.

So UrBackup decided yesterday that it’s time for a full backup on my laptop. I have a lot of small files and just going over them and getting their file size and change date usually takes half an hour. This is one reason why the incremental backup in UrBackup is so great. Because it does not have to go over all files.

Anyways after 8 hours it was finally finished and the server was happily sorting the files into the database (much faster now – see previous entry).
Then it basically started the full backup again! It couldn’t copy the list of all transferred files to the right location because all the space on the drive where these lists are saved was used up by temporary files. Now you probably shouldn’t store your temporary files on the same drive as you save e.g. the UrBackup database (as it can fill up often). But this is nevertheless a problem, which I fixed.

Now it’s backuping again. 4 hours and going. Hopefully there will be no other error at the end.

Keep in mind that this is not a serious bug, as the full backup was complete and no data was lost and no successive errors occurred. The bug only causes the backup to be worthless for successive incremental backups (they have to backup more). It also does only occur if you have less space in the temporary folder then the files which are backed up and if your temporary folder is on the same drive as your application data. This is not a good idea anyways.

UrBackup is a highly threaded application. Currently, for every client a server has approximately 5 threads are started. Additionally there is a thread pool for requests to the webinterface.
Some people would claim this is a difficulty in itself, because managing resources can be difficult the more threads you have. Anticipating this I built UrBackup using message passing and the implicit synchronization it provides.
Partly because of this I didn’t have that much problems.
What is a problem is how to tear down all those threads again. One solution which is currently “in use” is to let the operating system handle that. Let me elaborate on this:

The assumption is that you designed your application in a way that it can be forcefully stopped at any time. You should do that for every application that saves some kind of data, because nobody can guarantee for any length of time, that your procedure which stores data will not be interrupted and not started again. This interruption can be caused by a shortage of memory, a user killing the process or a power outage/hardware failure killing the computer.

Now designing an application like this can be hard. For example you write some settings to a file. You cannot write it directly there, because this write could be interrupted, causing an invalid settings file. The common pattern in such a case is to write the settings to a temporary file and to rename that file to your settings file. This can be done because the POSIX standard for filesystems defines that the rename operation is atomic – your filesystem guarantees that the rename either happens to the whole file or not at all (There are some different interpretations of this standard, however, which caused some issues e.g. with btrfs).

Now doing this with a lot of data is of course not efficient enough. You should use some library which does that for you. UrBackup uses sqlite (Which is an embedded database).

So you painstakingly did everything in a way that your program can be stopped at any time and no data is corrupted, or too much data lost. Do you still need to shut down your program gracefully? Assuming that only data is lost, that can be regenerated: No.

You can approach the problem from the other side as well. Assuming that you have time t during shutdown to write out data so nothing important gets lost you have to write your program in way that only so much data accumulates that you can still save it within t. Because sometimes you don’t now what t is or how long it is going to take to save your data you have to save important data as fast as possible. This means that the data which you would write at shutdown is not important – saving it is not necessary: You do not need save anything at shutdown.

So if every application does something like that, why does e.g. Windows not encourage the users to turn off the computer by removing their power source?

The first point here is of course that not every application does that. The second point is that sometimes you do not want them to do it: E.g. your laptop disk spins down. The application saves a little bit of data: The disk has to spin up again, in order to save the data.

The third and most important point is that often it cannot be guaranteed, that with one hundred percent probability the data is not corrupted by something like this. This can be the disk’s firmware/hardware fault, the filesystem driver’s fault or your programs fault. Do you still remember Windows laboriously checking your filesystem after you shut it down the hard way (It still does it some times)? This was in order to restore the integrity of the filesystem. Now the filesystems have gotten better at guaranteeing that integrity after a forcefull shutdown. But not with onehundred percent probability. It could guarantee it, but only with a performance penalty (that’s why it’s not done).

UrBackup has the same problem. It is only with a probability close to onehundred percent guaranteed that it’s database is not corrupted after a shutdown. There would be an option to further reduce this very small probability – with a huge performance penalty.

Of course the opportunities of such a corruption occurring should be minimized. Which means when possible one should not rely on the assumption that your program can be interrupted at any time.

That means for UrBackup, that I will have to sit down and work on a clean shutdown at some point in the future.