Sysadmin

(LinuxWorld) — We all make careless errors. Some of you may want to pretend you're immune, but I am certainly unashamed to confess that I am often my own worst enemy when it comes to computing.

My most recent disaster of foolishness occurred while setting up my system to compare the two journaling file systems Reiserfs and XFS. Somewhere along the line, I decided to convert some partitions on my workstation from ext3 to Reiserfs in order to gain some speed on those partitions. Reiserfs is faster than ext3 in part because of its clever design, and in part because it only journals metadata. ext3 is usually slower than Reiserfs even when you configure it to journal only metadata.

Unlike Reiserfs, ext3 can also journal data. This makes ext3 preferable for partitions where data integrity is paramount. Fortunately, Linux makes it very easy to mix file systems, so you can have your speed and not worry about anyone eating your data, too.

I still don't know quite how it happened, but I somehow backed up the wrong partition when I attempted to backup the /var partition. When it came time to restore the /var directory tree to the newly formatted Reiserfs partition, I discovered there was nothing to restore. As luck would have it, I had no other backups of the /var partition, so I couldn't even resort to using an old set of files.

Let me see if I can impress upon you how major a disaster this can be for a Debian system. There are a number of directories in any Unix system that will cause you trouble if you lose them, such as the /etc directory, where most of your configuration files reside (the /usr/share tree is another example).

When it comes to Debian in particular, the /var/lib/dpkg directory is an Achilles heel equivalent to the set of INI database files in OS/2, or the registry files in Windows NT, Windows 2000 or Windows XP. (The Windows 9x registry is also a weak spot, but it hardly matters since just about any damage can make a Windows 9x system unusable.)

To put this in its proper perspective, I've had to reinstall OS/2 and Windows more times than William Shatner has attempted to resurrect his acting career, and only because I had damaged INI files or the Windows registry. In comparison, I've corrupted whole Linux file systems (thanks to bugs in experimental kernels), and wiped out sensitive trees like the /etc or /usr/share trees mentioned above. I've upgraded all of my hardware multiple times, including all the disk drives. Yet, in spite of all these disasters and changes, I'm still running the very same installation of Debian I began with years ago.

(In case you're wondering how one keeps running the same installation with new drives, one could always use various backup systems to save and restore the OS. I took the fast and lazy route, however. I simply added the new drive to the machine along with the old one, copied the old installation to the new drive, and then removed the old drive.)

This time, however, the outlook seemed grim. The heart of Debian is its package system, and Debian stores all of its most important information about packages in the /var/lib/dpkg directory which was now gone. Most important of all is the /var/lib/dpkg/status file. I searched the Internet for tips on how to recreate this file from scratch. I found many instances of people wanting to know the same thing, but I found no answers. I came up with a couple of my own.

Three lessons learned

I've learned my lesson now, and that is to backup this file on a regular basis. However, in case you haven't learned this yet, here are the key things to remember, and two methods of restoring your /var/lib/dpkg directory from scratch without having to start from square one. (Granted, it's somewhat like starting from square three, but that's the price one pays for not performing regular backups.)

1. As I implied above, the /var/lib/dpkg/status file seems to be the most critical file. Debian makes a backup called "status-old" automatically. If you can find this, or a copy of it or the original status file somewhere, then restore it. If you can't, that should hammer home the most important tip of all. Backup this file, if not the whole directory from /var/lib/dpkg on down, on a regular basis, and keep the backup safe.
2. Okay, so you don't have the status file anywhere, and you don't have a backup. There's still hope. The following solution can be a bit messy, but it can save you oodles of time since you won't have to re-install as many packages to inform Debian of their existence. Here's what you do: Assuming you are like me, and have been upgrading Debian for years over the Internet, you don't have a recent Debian CD.

   If you have another Debian system that is reasonably similar to the one where you lost your /var directory, you can try restoring your /var tree from that system. If you have more than one Debian system from which to choose, pick one that is most likely to have the same software installed or a subset of the software you have installed on the system you are trying to recover. This is how I restored my system, and I had it back to normal in less than an hour.

   It's not the end of the world if the system you are copying from has a superset of the packages you have installed on the broken system, but it will more likely lead to minor annoyances later on. Your restored system will think it has more software on it than it really does. When you install future packages, they may install without noticing you don't have the proper dependencies met. Even these problems are not insurmountable, but you can keep your stress level down by picking a backup system with a subset.

3. As a last resort, assuming you have absolutely nothing you can use as a makeshift backup, then here's a method I tested for rebuilding the /var/lib/dpkg tree from nothing. It worked fine on my test system based on the unstable branch of Debian, but beware. No two Debian systems are 100 percent identical, so no doubt you may run into a slightly different set of requirements and problems on yours.

   Just read the error messages you encounter and be prepared to be flexible if one or more steps give you trouble. I can almost guarantee there's a solution to every glitch, so keep at it. Remember that not all of the error messages you see are important. Some, such as the complaints you'll encounter about insufficient information about libc6 in my solution, can be ignored, since they'll go away by the end of the procedure.

First, recreate the following directory trees under /var, at minimum (if you can recreate more of your /var tree, then by all means, do so.) Obviously, you want to perform all the following tasks as root or whatever other privileged account you use for system administration.

/var/cache/apt/archives
/var/cache/debconf
/var/log
/var/lib/dpkg/info
/var/lib/dpkg/parts
/var/lib/dpkg/alternatives
/var/lib/dpkg/methods
/var/lib/dpkg/updates

Then create this status file:

/var/lib/dpkg/status

Put the following text into the status file. If your system has a different version of libc6, then substitute that version number in the example I provide below. Make sure to press return after typing in the version number, because dpkg needs to see a newline there. This is not a complete entry for libc6, it's just enough to get you going with a minimum of effort.

Package: libc6
Status: install ok installed
Version: 2.3.1-9

Now execute the following commands, one at a time. The first time you try to execute apt-get dist-upgrade, it will download all the packages it needs but fail to install any of them. That's fine. We just want to get these packages downloaded for now. Then change to the /var/cache/apt/archives directory and run the dpkg installation commands. It's rather amusing to see dpkg complain that dpkg isn't installed, but it obviously functions well enough to get your system into a manageable state where you can install it again.

dpkg --clear-avail
apt-get update
apt-get dist-upgrade
cd /var/cache/apt/archives
dpkg -i libncurses*
dpkg -i perl-base*
dpkg -i libstdc++*
dpkg -i dselect*
dpkg -i dpkg*

This brings you almost to a point where you can proceed without programs complaining about what you're doing. The last complaint you need to get rid of is the one about the insufficient information about libc6. Just reinstall it to update your status file with the missing information.

apt-get install --reinstall libc6

Finally, re-run the dist-upgrade command. This time it will work and install all the base packages for Debian.

apt-get dist-upgrade

Once you get to this point, you can re-install the rest of the software you want apt-get/dpkg to know about in a variety of ways. The dselect package manager is ugly and unfriendly, but it is quite effective for this purpose. If you prefer, you may be able to start up an X session (the software and configuration files are there, it's just that Debian doesn't officially know it.) Then install the friendly and powerful graphical Debian package manager synaptic and reinstall the software that way. Yes, all of this software is already installed, but what you are doing is filling up your /var/lib/dpkg/status file with current information about your system.

Granted, this last procedure is about as close to reinstalling Linux as one can get without duplicating the effort required to reinstall Windows or OS/2. However, in my experience, it's not nearly as time-consuming. When you lose the Windows registries, you lose everything, including driver information and application configuration data.

When you lose your /var/lib/dpkg directory, you only lose information about the status of what is and is not installed on your system. It doesn't wipe out your system or application configuration files, your applications, or the system's knowledge about the kernel and its modules (drivers). You won't have to reconfigure your applications, X11, the network, hardware drivers, and any of the other tedious tasks you might have to perform when re-installing Windows from scratch.

Be careful. Make backups. If you slip up, I hope the above tips will help you recover.