home of the madduck/ blog/
Speed up dpkg on older systems

My main desktop is 7 or 8 years old and has never been reinstalled. dpkg has become so slow that I started to shy away from using it, e.g. to purge packages. Not good.

I always suspected that it was due to /var/lib/dpkg/status, the ever-growing dpkg status "database"^W plain-text file, which the programme had to scan multiple times per run, and I sought a means to compact it, and by that I meant:

dpkg --compact

or similar to remove the stanzas for purged packages. There is --forget-old-unavail, but that only cleans out uninstalled packages that are no longer available (I think), whereas I'd really rather get rid of knowledge of all uninstalled packages. Who needs that information if you don't use --get-selections and --set-selections?

While I was still huffing about the absence of such a command-line option, and inspecting a bug report about auto-cleaning the status database of purged packages, Gerfried Fuchs got to the matter and suggested to run (needs dctrl-tools and sudo installed):

# first ensure a backup in /var/backups
sudo /etc/cron.daily/standard
grep-status -! -FStatus 'purge ok not-installed' > /tmp/dpkg.status
  && mv /tmp/dpkg.status /var/lib/dpkg/status

That shrunk the file by about 10%, which didn't impress me, and so I continued to complain…

… until it struck me that the speed increase from removing what must have amounted to hundreds of purged package stanzas was enormous.

Highly recommended — until dpkg does better housekeeping. Looks like that will be the case with the next version. According to Guillem Jover, the dpkg maintainer caring about this right now, none of the above will be needed in the near future, since dpkg will be tidier, and also do the spring-cleaning on the first run.

Update: Matt Zimmerman suggested that the status file is hardly the problem. Instead, he purports from experience that the many small files in /var/lib/dpkg/info, which fragmented across the disk are the true hog:

mizar:~# echo 3 > /proc/sys/vm/drop_caches
mizar:[~] time cat /var/lib/dpkg/status >/dev/null
cat /var/lib/dpkg/status > /dev/null  0.00s user 0.01s system 4% cpu 0.270 total
mizar:[~] time sh -c 'cat /var/lib/dpkg/info/*.list' >/dev/null
sh -c 'cat /var/lib/dpkg/info/*.list' > /dev/null  0.06s user 0.83s system 9% cpu 9.471 total

I will investigate this at some point, but not anytime soon.