Debian-Ubuntu weakness to filesystem corruption recovery

I really love Debian based distribution, namely, Debian itself and Ubuntu, but there is something wrong… The weakest part is located in /var/lib/dpkg/.

Well, yesterday my /var partition got corrupted and because of that I had to reinstall my whole system. This could (and should) have been avoided. But I had no choice left.

Let’s read more and you will see why I had to reinstall the whole system and how it could have being avoid.

I usually set up my system with a : * / partition (1.5G) hosting all that is necessary to run a linux system (/bin and /sbin) * /usr (4G), this is where all extra distribution packages will be stored * /home (10G). Where all my personnals data, computing work and programs are stored. By using a separate /home partition you can safely install a new Unix system and keep all your bookmarks, user specific conf … * /tmp (actually I’d be better off aggregating / and /tmp partition) * /var : the place where services are stocking their data, I will go through it later on.

The /var directory is a really precious one when running a server, this is the place where mysql keep his datas, postfix (and other mailing soft) his mails.

Debian based system creates a repertory named /var/lib/dpkg. This is the place where the system is going to keep it’s precious datas.

A good overview of How-to maintain a Debian System explain that the list of installed packages is located in /var/lib/dpkg/status file and that This file must not be corrupted, or otherwise your system is HOSED as this is the main database for the package manager software: dpkg. Hopefully, because Unix system are often used for production, a back up of this file is made, named status-old.

But because this file is too important for the system, you better take double security measure in order to avoid a disaster. That is the reason why a cron job save this file on a daily basis, along with /etc/password, /etc/group, /etc/shadow.

Well great then! Even though you delete /var/lib/dpkg, you are still out of the mess… Well, not really. I doubt that this backup is made in order to help people inadvertenly erasing /var/lib/dpkg. I guess this is rather to provide necessary file to recover your system.

But then, what happens if the whole /var get corrupted?

YOU JUST GET F****D !!!

Yep, that is just like keeping your second keyring inside your house, if you lose the first keyring, you don’t give a damn about second keyring that you will find in the place you can’t access, aren’t you?

I believe that having a separate partition for /var is a save way to protect your system, basically, if I get troubles on any other partition than /, I should be able to at least get a running system and being able to set back up the rest (even though precious datas might have disappeared).

CONCLUSION:

Because of this bad experience I had, I’m not going to keep debian making backups in the same place. /var/backups is a pretty small directory, therefore I’m going to keep the backups in /root and change in /etc/cron.daily/standard:

bak=/var/backups

by

bak=/root/backups

I don’t know what you guys think of this, but I would be really glad to get your comment and see if any other alternatives could be done.

The best would be to at least be able to rebuild the /var/lib/dpkg and /var/cache/apt trees so the system can run without troubles.