IPv6 with Debian

Debian seems to be ready for IPv6 (although there are still some problems). In less than a day, I put a few of my machines online and joined the Internet of the future. In the following, I’d like to share with you how I did it.

Configuring the packet filter

The first thing I did was to configure the packet filter on each host appropriately. Unfortunately, this is harder than it should be, because — to quote one of the netfilter developers — “when ip6tables was conceived, someone had a big bad brainfart”: rather than adding IPv6 rules to your existing iptables ruleset, you have to create a new ruleset, duplicate all chains, networks, hosts, and individual rules, and maintain the two in parallel. Even though there are efforts of unification on the way, I speculate it’ll take another couple of years until PF_INET6 will be fused into PF_INET and one will be able to do sensible cross-address-family packet filtering with Linux. Since I’ve recently started to look (again) at pyroman, maybe the most logical way forward would be to extend it to write both, IPv4 and IPv6 rulesets from its knowledge about the hosts and networks you configured.

Anyway, we want to get stuff working now! Thus, let’s configure ourselves a packet filter. (Almost) all IPv6-related filtering must be configured via ip6tables (read on further down about IPv6 in IPv4 tunneling, the reason I said “almost”). The following is a simple default ruleset to start with, which I put into /etc/network/ip6tables to load with ip6tables-restore:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:in-new - [0:0]

### INPUT chain

# allow all loopback traffic
-A INPUT -i lo -j ACCEPT

# RT0 processing is disabled since 2.6.20.9
#-A INPUT -m rt --rt-type 0 -j REJECT

# allow all ICMP traffic
-A INPUT -p icmpv6 -j ACCEPT

# packets belonging to an establish connection or related to one can pass
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# packets that are out-of-sequence are silently dropped
-A INPUT -m state --state INVALID -j DROP
# new connections unknown to the kernel are handled in a separate chain
-A INPUT -m state --state NEW -j in-new

# pass SYN packets for SSH
-A in-new -p tcp -m tcp --dport 22 --syn -j ACCEPT

# log and reject everything else
-A INPUT -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[INPUT6]: "
-A INPUT -j REJECT

### OUTPUT chain

# RT0 processing is disabled since 2.6.20.9
#-A OUTPUT -m rt --rt-type 0 -j REJECT

# allow outgoing traffic, explicitly (despite chain policy)
-A OUTPUT -j ACCEPT

### FORWARD chain

# RT0 processing is disabled since 2.6.20.9
#-A FORWARD -m rt --rt-type 0 -j REJECT

# disallow forwarded traffic, explicitly (despite chain policy)
-A FORWARD -j REJECT

COMMIT

Note that this recipe is pretty much unusable on pre-2.6.20 kernels due to their broken implementation of stateful connection tracking.

The ruleset should be fairly obvious, but you might wonder about my use of REJECT and allowing all ICMP — after all, you’ve heard for the past 30 years that ICMP is a “bad hacker protocol”, and Internet security is no domain for being nice to people, so to prevent any information disclosure, you should DROP connections, not let people know that they’re simply not allowed.

Well, to hell with all that! I don’t see a single reason or attack vector that is foiled by DROP or disallowing ICMP. In fact, it’s just security by obscurity, and mighty inconvenient at the same time. ICMP is also much more important with IPv6 than with IPv4 (it replaces ARP, for instance), and it’s actually useful to be able to ping hosts, or get back informational messages on why something failed. Finally, rejecting traffic rather than dropping it doesn’t suggest to a hacker that something’s hidden here.

Then there is RFC 4890, which almost made me puke. This document is part of the reason why I say: let’s fix problems in the kernel, rather than shielding them with unreadable and unmanageable rulesets!

Setting system parameters

The /proc/sys/net/ipv6 filesystem exports a number of parameters that you might want to set. The Linux IPv6 HOWTO explains all available parameters, so let me just show you the ones I set in /etc/sysctl.conf and load with a call to sysctl:

net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.accept_ra = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.all.autoconf = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.all.accept_ra_defrtr = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 0
net.ipv6.conf.all.accept_ra_pinfo = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.all.forwarding = 0

Obviously, for gateways, you want to enable forwarding, and on machines where you want to use autoconfiguration, those accept_ra* entries ought to be 1 accordingly.

If you’re curious, the default values (the first seven lines) set the defaults for new interfaces, while the all values override the values for all interfaces. I am not sure whether this is a permanent overwrite, or just overwrites all existing interfaces, which is why I simply set the parameters for both.

Unfortunately, this does not seem to work. It seems as if writing to all does not affect existing interfaces, as documented in this thread on the linux-kernel mailing list.

At this moment, I suggest that you add pre-up headers to the iface stanzas of all interfaces to be sure:

pre-up echo 0 > /proc/sys/net/ipv6/conf/$IFACE/autoconf
pre-up echo 0 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra
pre-up echo 0 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra_defrtr
pre-up echo 0 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra_pinfo
pre-up echo 0 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra_rtr_pref

Getting connected

If you already have an IPv6 address, you’re basically ready to go, but may want to read further down on how to connect your local network to the IPv6 Internet as well. If you are searching for a provider, have a look at the list of providers with native IPv6 connectivity over at sixxs.net.

If you are reading up to here, I assume you are connected to the ‘Net with IPv4. There are two ways for you to move towards IPv6: 6to4 or by way of a tunnel provider. A Kiwi website explains how to setting up 6to4 connectivity, and thus I will concentrate only on the tunnel approach. Even though everyone can set up 6to4 in a breeze without any accounts or waiting, there are a number of security considerations, it’s pretty ugly to debug (due in part to asymmetric routing), and makes your life unnecessarily difficult when all you have is a dynamic IP that changes from time to time. If you are stuck behind a NAT gateway, you cannot use 6to4 either. Thus, I prefer the tunnel approach.

With the tunnel approach, IPv6 packets are wrapped up in IPv4 packets on your host, and sent to the IPv4 address of your tunnel provider, who has native IPv6 connectivity. The tunnel provider unwraps your packet and shoves the contained IPv6 packet onto the backbone. The IPv6 address you used as source address is routed to the tunnel provider, so any replies arrive at their machines, where they’re again wrapped into IPv4 packets and sent to your (publicly-accessible) IPv4 address. Those IPv4 packets specify payload type 41 (“ipv6”), which is why we need -p ipv6 -j ACCEPT rules in the IPv4 iptables ruleset (see below).

There are a few tunnel providers out there. I chose SixXS and have not regretted my choice. I shall thus assume that you do the same: sign up for an account right now, so that you have it by the time you finished reading this document! SixXS works on a credit system: tunnels and subnets cost credits, which you can accumulate by maintaining your tunnels properly. This ensures that everyone can play around, but to do more advanced stuff, you need to first display competence with the basic concepts.

Tunnel types

Your first step with SixXS will be to request a tunnel. SixXS offers three types of tunnels:

• static tunnels, for those with static IPs,
• heartbeat tunnels, for those with dynamic IPs, and
• AYIYA tunnels, for those behind NAT gateways.

Each of these tunnels have advantages and disadvantages, as everything does: the first two types of tunnels use IP protocol 41 packets to encapsulate the IPv6 packets. As such, there are security considerations involving the impersonation by spoofing, and all upstream firewalls must let protocol 41 pass. AYIYA addresses these problems by using signed packets, but that solution comes with extra computation overhead and smaller MTUs.

I suggest to use the first type of tunnel that fits your situation. Debian’s aiccu package can take care of heartbeat and AYIYA tunnels for you, and it can even set up static ones.

During registration, you will also need to choose a “PoP”, which stands for “Point of Presence”. If your country only has a single PoP, that’s the one you will end up using (unless you have a good reason for another one), but if there are more options, I strongly suggest that you go through the list of PoPs and select the one with the best roundtrip time and lowest latency from your location! Note that you must answer ping requests (ICMP echo-request) from the PoP you chose, or else the tunnel will not be created.

Other tunnel brokers include HE, which give you up to four tunnels and subnets within minutes, but they only support proto-41 tunnels (“static tunnels”), which do not work everywhere.

Setting up the tunnel

Once your tunnel request gets approved, you’ll get a /64 prefix, in which you only use two addresses: the PoP will configure the :1 address and you need to configure your host to use the :2 address on the tunnel interface. You’ll also be told the IPv4 address of your PoP “endpoint”.

Joey Hess taught me that aiccu can set up the interface for you, using the data it queries from the SixXS registration (TIC) server. I tried it, and it works. However, I prefer the pure ifupdown approach, as it makes things explicit and allows me to use the hooks for stuff like loading the packet filter. So in my /etc/network/interfaces, you can find (remember that these are fake addresses and you need to use your own prefixes and PoP endpoint address):

auto sixxs
iface sixxs inet6 v4tunnel
  endpoint 10.111.222.33
  address 2001:db8:ff00:3b::2
  netmask 64
  gateway 2001:db8:ff00:3b::1
  ttl 64
  pre-up ip6tables-restore < /etc/network/ip6tables
  up ip link set mtu 1480 dev $IFACE
  up invoke-rc.d aiccu start
  down invoke-rc.d aiccu stop

Make sure to read about MTU values of the tunnel and adjust the 1480 value in the above to your tunnel settings and ISP connectivity.

The last two lines are only needed for heartbeat tunnels and assume that you have configured aiccu appropriately.

Also, when using /etc/network/interfaces, make sure to set noconfigure true in /etc/aiccu.conf, or else aiccu will bring up a duplicate/additional interface. If your version of aiccu does not honour this configuration option, you can set ipv6_interface sixxs instead: if you tell it to configure the same interface as specified in /etc/network/interfaces, it will actually execute all the same commands (which will fail), but won’t report any errors.

For AYIYA tunnels, the following can be used, if you prefer to be able to bring up the interface with the usual ifup command, rather than having it appear when the daemon starts.

auto sixxs
iface sixxs inet manual
  pre-up invoke-rc.d aiccu start
  up sleep 1
  up ip link set mtu 1280 dev $IFACE
  post-down invoke-rc.d aiccu stop

It is also a good idea to prevent aiccu from starting at boot when using this method:

update-rc.d aiccu disable

Allowing proto-41 traffic

Unfortunately, these methods are likely to fail, because your regular IP packet filter (iptables, without the 6) doesn’t let those encapsulating IPv4 packets pass, unless you tell it to; you probably want to do this early on in the chain, and also limit it to the tunnel peer, so in my case, I use the following rule:

iptables -I INPUT -p ipv6 -s 10.111.222.33/32 -j ACCEPT

For AYIYA, you need to open port 5072, either for UDP, TCP, or SCTP, depending on how you configured it. Also have a look at this FAQ entry on what a firewall needs to pass. If it still doesn’t work, you have an upstream packet filter that needs some of those holes poked. Good luck.

In most situations, the FORWARD chain does not get such a rule, since the tunnel terminates at the gateway, which routes to a native IPv6 network, or another tunnel. Allowing tunnels through a gateway is almost always a bad thing, as it would allow undetected and untraceable traffic from compromised boxes in the local network. The OUTPUT chain also does not need such a rule, if you have configured stateful filtering properly.

Now bring up the interface and verify the connection:

# ifup sixxs
# ping6 -nc1 2001:db8:ff00:3b::1
PING 2001:db8:ff00:3b::1(2001:db8:ff00:3b::1) 56 data bytes
64 bytes from 2001:db8:ff00:3b::1: icmp_seq=1 ttl=64 time=74.0 ms
[...]
# ping6 -nc1 ipv6.aerasec.de
PING ipv6.aerasec.de(2001:a60:9002:1::184:1) 56 data bytes
64 bytes from 2001:a60:9002:1::184:1: icmp_seq=1 ttl=55 time=91.5 ms
[...]

Welcome to the Internet of the future!

Setting up an IPv6-capable gateway

Your IPv6 connection works, but it’s limited to a single address, and you do not get to specify the reverse DNS PTR record for it. Since the concept of NAT is mostly absent from IPv6 (thanks! thanks! thanks!), you will not be able to connect any other hosts to the IPv6 network. If your local network has a few hosts behind a gateway, you will need to request a subnet from SixXS and configure your gateway (which has the tunnel connection) appropriately. Don’t worry, this is not really very difficult.

First, request a subnet for your tunnel from your PoP via your SixXS homepage. Once approved, you will get a /48 prefix for your own use: 2^80 — 1.2 heptillion addresses which are yours to assign to every dust particle in your office or home, if you so desire.

The way I set it up is to add the first of these addresses to your internal interface on the gateway, by adding the following two lines to the interface’s stanza in /etc/network/interfaces; you will need the iproute package installed (which you should be using for everything network-related anyway; remember to substitute your own prefixes):

up ip -6 addr add 2001:db8:ff12::1/64 dev $IFACE
down ip -6 addr del 2001:db8:ff12::1/64 dev $IFACE

Instead of bringing the interface down and up, just run ip -6 addr add <your prefix> dev eth0. Note the use of the /64 prefix instead of the /48 that got assigned, leaving only 20 pentillion addresses. Oh no! The reason for this is buried in the specs: basically, /48 is a site prefix, but individual networks should not be larger than /64, which is the prefix length of links in the IPv6 domain.

The /64 prefix is only one of 65536 different /64 prefixes you can use from your /48 prefix. Since it’s unlikely that you’ll use them all, it’s a good idea to route unused ones to an unreachable destination, such as the loopback interface, which conveniently causes packets to any addresses outside the used /64 networks to be answered with ICMP destination network unreachable. You could route them to the special unreachable target instead, which would cause host unreachable messages, but the following is more explicit (again, remember to use your own prefix in place of the one used in this document):

up ip -6 route add 2001:db8:ff12::/48 dev lo
down ip -6 route del 2001:db8:ff12::/48 dev lo

Enabling forwarding

Now is also a good time to enable IPv6 forwarding, e.g. like so:

# echo net.ipv6.conf.all.forwarding = 1 >> /etc/sysctl.conf
# sysctl -p

Obviously, you will also need to change the policy on the ip6tables FORWARD chain. For now, let’s just set it to accept all traffic between the local network behind eth0 and the Internet behind eth1. You should later create a proper ruleset, though! (remember to use your own prefix)

# ip6tables -I FORWARD -i eth0 -o eth1 -s 2001:db8:ff12::/64 -j ACCEPT
# ip6tables -I FORWARD -i eth1 -o eth0 -d 2001:db8:ff12::/64 -j ACCEPT

Advertising addresses on your local network

The final step is to spread the love to your local network. Refrain from selecting addresses from your subnet and assigning them to the local hosts, or wondering how to configure the DHCP server, because IPv6 does it differently: your gateway will advertise its routes (which includes a default route) to your network, and each host will pick an address based on its MAC address (unless it already has an EUI-64 address assigned. This all happens automagically, at least with current Debian and Windows machines.

On the gateway, you need to install radvd and simply tell it which prefix to use on which interface. My /etc/radvd.conf looks like this, and I won’t explain it (remember to use your own prefix, not the one used in this document):

interface eth0
{
  AdvSendAdvert on;
  prefix 2001:db8:ff12::/64
  {
  };
};

Note again how we advertise a /64 network rather than the /48 we “own”. You cannot advertise smaller networks if you want automatic configuration to work, and you should not use networks larger than /64 in any case. If 2^64 addresses are not enough for you, I trust you’ll be able to figure out how to advertise another of your 65536 /64 prefixes in the /48 subnet to appropriate hosts.

Restart radvd and run over to another host to witness how it automagically gets connected to the IPv6 network by scanning /var/log/kern.log and watching the output of ip -6 addr and ip -6 route. Try ping6ing from there! Find the dancing turtle! It should all work.

If you don’t like the automagic aspect of this, look into stateful configuration, using DHCPv6, as provided by dibbler-server and wide-dhcpv6-server.

Choosing source addresses

Even though you have obtained a /48 network and configured the gateway appropriately, connections originating from the gateway itself will have the tunnel endpoint IP as source address. This is because Linux picks the address of the outgoing interface when it sends packets, and we only slapped the address from our own subnet onto the LAN interface. However, if the firewall answers to 2001:db8:ff12::1 (or whatever address you gave it from your prefix space), it ought to use that address for sending, too.

Unfortunately, there’s no guaranteed way to achieve this, due to RFC 3484, which is explained nicely on this page.

The best solution is to add 2001:db8:ff12::1/128 as last address to the external interface, which will cause Linux to use it in most cases. When it doesn’t, you ought to just ignore it (remember to use your own prefixes instead):

iface sixxs inet6 v4tunnel
  […]
  up ip -6 addr add 2001:db8:ff12::1/128 dev $IFACE
  down ip -6 addr del 2001:db8:ff12::1/128 dev $IFACE

Here is a thread on the issue on the SixXS forum.

Resolving names

Take note of the IPv6 address of each host. There’s a way to determine it given the host’s MAC address, but this is easier (ipv6calc is also useful). You might want to let your local DNS server know by adding AAAA records in parallel to the existing A records, and possibly even adding PTR entries.

If you’re serious about IPv6, you can tell SixXS to delegate reverse lookups for the IPv6 addresses to your DNS servers, but you ought to refrain from polluting the DNS namespace. The dig tool and its +trace option are awesome to debug and analyse reverse delegations:

dig +trace -x 2001:41e0:ff1a::1

Note that bind9-host provides an improved host tool, which fetches all kinds of information about names, not just the one single information configured as default:

% host pulse.madduck.net
pulse.madduck.net has address 130.60.75.48
pulse.madduck.net has IPv6 address 2001:41e0:ff1a::1
pulse.madduck.net mail is handled by 99 b.mx.madduck.net.
pulse.madduck.net mail is handled by 10 a.mx.madduck.net.

% host 2001:41e0:ff1a::1
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.1.f.f.0.e.1.4.1.0.0.2.ip6.arpa
domain name pointer pulse.madduck.net.

Oh, and if you’re really that curious about how IPv6 addresses are computed from MAC addresses, read RFC 2464. Basically, given a prefix 2001:db8:ff1a:: and a MAC address aa:bb:cc:dd:ee:ff, the resulting IPv6 address is obtained by:

1. inserting ff:fe into the middle of the MAC address to yield aa:bb:cc:ff:fe:dd:ee:ff;
2. removing the odd colons to yield aabb:ccff:fedd:eeff, the EUI-64;
3. flipping the second lowest bit of the first octet to yield a8bb:ccff:fedd:eeff; this marks the EUI-64 globally unique.
4. concatenating the prefix and this result to get 2001:db8:ff1a::a8bb:ccff:fedd:eeff.

Other (easier) ways to determine the local component of an IPv6 address are:

• look at the output of ip -6 addr and find the address starting with fe80:: on the interface you want to connect to the IPv6 network; following the two colons is the interface’s EUI-64.
• use ipv6calc or sipcalc.

If you find your (Windows) IPv6 addresses changing all the time, you might be faced by “privacy features”.

Remaining issues

Even though my IPv6 connectivity works, I have two remaining issues.

Mapping names to laptops

Laptops generally have two interfaces, one with a cable, and the other wireless. Both of these interfaces will have separate MAC addresses, and by extension, the laptop will have different IPv6 addresses depending on how it is connected to the local network.

I want to be able to connect to laptops without knowing the medium they use to connect to the network. Unfortunately, there seems to be no feasible way. The solutions I see are:

• override the MAC address of one interface with that of the other, which is going to cause bgi problems in the case when the laptop (accidentally) gets connected to the same network twice;
• add both IPv6 addresses as AAAA records to the laptop’s DNS name, which will cause random delays when connecting as the resolver may return the currently inactive address first;
• set up mobile IPv6, e.g. by following this Mobile IPv6 how-to, which would allow accessing the laptop uniformly, no matter where in the world it is. Unfortunately, Debian’s support for Mobile IPv6 is severly lacking at time of writing. Also, Yves-Alexis Perez notes that this how-to is horribly outdated and promised to tend to it Real Soon Now™.

The second solution works for me for now, but I am interested in the third.

In response to this document, Andreas Henriksson has suggested the replace the stateless configuration (radvd) with stateful configuration, using DHCPv6. I have yet to investigate this option.

Jeroen Massar suggests to unite cable and wireless into a bridged interface, which seems like a very good idea.

Kernel versions and stateful connection tracking

Even though the default kernel of Debian etch, our current stable release, speaks IPv6, I cannot recommend it for deployment, as the 2.6.18 kernel does not support proper stateful connection tracking for IPv6, and thus makes it impossible to firewall hosts in a sensible manner (I always add local packet filters to all my hosts, and if only to guard against the situation when a user installs a malicious programme to listen on a high port). Of course, it is possible to configure a packet filter statelessly in an acceptable manner once you know the use case, so do with this information what you wish; I prefer to stay general for now.

A remedy exists, however: Debian “etch” 4.0r4 adds the so-called etch-and-a-half kernels, e.g. linux-image-2.6.24-etchnhalf.1-686. The 2.6.24 kernel seems to support stateful connection tracking for IPv6. Alternatively, you could use the 2.6.24 linux kernel packages on backports.org.

Xen and IPv6

One drawback of switching to 2.6.24 is that you cannot run a dom0 on that machine any longer, so by practical extension, you cannot connect it to the IPv6 network with a packet filter in place. Supposedly, running 2.6.24 instances on a 2.6.18 dom0 is reported to work, however.

Debian “lenny” 5.0 fixes this: it comes with a patched 2.6.26 kernel and can be used for both, as dom0 as well as regular IPv6 citizen.

Credits

Thanks to Bernhard Schmidt, William Boughton, and Jeroen Massar, and everyone on #ipv6/irc.freenode.org for their help over the past few weeks, and all those who fed back comments in response to this document!

Posted Thu 01 Apr 2010 14:33:21 CEST Tags: blog debian ?documentation ?howto ?ipv6 ?networking planet-debian

Auto-subscribing to Debian bugs I file

It happens from time to time that bug reports I file receive attention, but I don’t notice because our bug tracking system still does not auto-subscribe bug submitters to their own bugs (see bug #37078 and bug #351856). I thus decided to take the matter into my own hands.

Just in time, before I started hacking this up myself, I found Justin Pryzby’s procmail recipies, which are installed to /usr/share/doc/devscripts/examples/bts_autosubscription.procmail by the devscripts package. The result is available in my mailfilter git repository. So far, I only auto-subscribe in response to bugs I file; Justin also auto-subscribes to bugs he manipulates via the control bot.

For completeness, I also wanted to subscribe to all bugs that I have submitted. This turned out to be easier than I thought, thanks to bts in the devscripts package; note the sleep instruction in the loop to prevent hammering the system (please remember to substitute your e-mail address twice):

bts select submitter:your.e-mail@add.ress \
  | while read bugno; do
      echo X-debbugs-autosubscribe: doit! \
        | sendmail -f your.e-mail@add.ress ${bugno}-subscribe@bugs.debian.org
      echo subscription to \#${bugno} sent
      sleep 30
    done

NP: The Dukes of Leisure: The Dukes of Leisure

Posted Thu 10 Sep 2009 22:06:11 CEST Tags: blog ?debbugs debian ?devscripts planet-debian

Speed up dpkg on older systems

My main desktop is 7 or 8 years old and has never been reinstalled. dpkg has become so slow that I started to shy away from using it, e.g. to purge packages. Not good.

I always suspected that it was due to /var/lib/dpkg/status, the ever-growing dpkg status “database”^W plain-text file, which the programme had to scan multiple times per run, and I sought a means to compact it, and by that I meant:

dpkg --compact

or similar to remove the stanzas for purged packages. There is --forget-old-unavail, but that only cleans out uninstalled packages that are no longer available (I think), whereas I’d really rather get rid of knowledge of all uninstalled packages. Who needs that information if you don’t use --get-selections and --set-selections?

While I was still huffing about the absence of such a command-line option, and inspecting a bug report about auto-cleaning the status database of purged packages, Gerfried Fuchs got to the matter and suggested to run (needs dctrl-tools and sudo installed):

# first ensure a backup in /var/backups
sudo /etc/cron.daily/standard
grep-status -! -FStatus 'purge ok not-installed' > /tmp/dpkg.status
  && mv /tmp/dpkg.status /var/lib/dpkg/status

That shrunk the file by about 10%, which didn’t impress me, and so I continued to complain…

… until it struck me that the speed increase from removing what must have amounted to hundreds of purged package stanzas was enormous.

Highly recommended — until dpkg does better housekeeping. Looks like that will be the case with the next version. According to Guillem Jover, the dpkg maintainer caring about this right now, none of the above will be needed in the near future, since dpkg will be tidier, and also do the spring-cleaning on the first run.

Update: Matt Zimmerman suggested that the status file is hardly the problem. Instead, he purports from experience that the many small files in /var/lib/dpkg/info, which fragmented across the disk are the true hog:

mizar:~# echo 3 > /proc/sys/vm/drop_caches
mizar:[~] time cat /var/lib/dpkg/status >/dev/null
cat /var/lib/dpkg/status > /dev/null  0.00s user 0.01s system 4% cpu 0.270 total
mizar:[~] time sh -c 'cat /var/lib/dpkg/info/*.list' >/dev/null
sh -c 'cat /var/lib/dpkg/info/*.list' > /dev/null  0.06s user 0.83s system 9% cpu 9.471 total

I will investigate this at some point, but not anytime soon.

Posted Tue 18 Aug 2009 11:54:48 CEST Tags: blog debian ?dpkg ?improvements planet-debian ?tips

Managing Debian packages with GNU arch

Manoj (thanks to Jaldhar, I finally know how to pronounce your name… it would be spelt “Manosch” in German and bears resemblance to the name of one of the greater German child book authors)… uh, so where was I?

Oh yeah, Manoj has a really interesting way to manage packages with GNU arch, which I have started to adopt for my packages. The approach gets rid of things like dpatch (yes!) and simply manages additional features and debianisation in separate branches. Read the last paragraph of this entry if you can’t be bothered with the rest.

Even though bazaar still has its rough edges, it’s a pleasure to work with the guys making it possible, and bugs have quick turnaround times. This is one of the reasons why I chose to adopt Manoj’s method together with bazaar for pkg-zope as well as pkg-mdadm. I have started to document the process on the pkg-zope wiki.

The only real drawback of Manoj’s method is that it has a steep learning curve. Thus, Mario (the other mdadm maintainer) and I had the idea to hold a live session on IRC in which to demonstrate the use of arch and allow people to ask questions.

We are meeting for this live demonstration and Q&A session tomorrow, Thursday 11 August 2005, at 19:00 hours GMT in the #pkg-zope channel on irc.debian.org, and you are cordially invited to join. I’ll also log the session and make it available online afterwards. Hope to see you there.

Posted Mon 09 Mar 2009 16:16:24 CET Tags: ?arch blog debian planet-debian ?vcs vcs-pkg

FOSDEM 2006

FOSDEM 2006 is over, I am now sitting at one of the weirdest airports I have seen so far: passages are intertwined and long, and it seems like you have to walk several hundred kilometres in circles before you finally end up at the place you ned to go to.

I gave a talk Saturday on my Ph.D. research. Having spent almost an entire week preparing it, which involved rethinking my topic, the approach, and fighting doubts several times, I was all too pleased that a new aspect of my research was dawning upon me just in time to change a lot of the slides around a bit on the plane from Zurich. Even though my goal is to work a lot on Debian, I think the question on which I really want to concentrate is along the lines of “how to get volunteers to adopt new models and methods.” Debian seems like a perfect environment in which to search for an answer, but I assume I shall also be looking around the Zope and Plone communities.

Unfortunately, my talk didn’t go as well as I had hoped. Part of it was on the Debian security situation, and having talked to a couple of people up front, and receiving even more feedback to my online personna, which apparently sometimes misrepresents my intentions, I took specific care to do with the delicate topic of Debian security with utmost care (and I think I succeeded; oh, and as a German I have a right to run-on sentences…). As a result though, I lost about 10 minutes of time, and additionally, I got confused about the length of my talk slot somewhere in the middle of the talk.

After giving an overview of my research idea, model, and approach, and the security situation, and just before coming to the exciting meat on possible extensions to Debian package management and the upload process which are geared to making it easier for others to contribute, h01ger kindly pointed out I had 10 minutes (not 25) left, a bunch of people walked into the room, generating noise and switching lights on and off, and on top of it all, OpenOffice crashed with the remainder of my slides — those that contained graphs and other means to convey the complex ideas I was about to introduce.

Everything seemed to be going wrong. I tried to give brief overviews of the ideas from the top of my head, but I didn’t manage to piece together all the details that made the ideas interesting in a Debian context, and how the approach differs from Ubuntu’s Launchpad and their new tool hct. So at the end of the talk I felt like I hadn’t persuaded anyone of anything. Oh well. I was down for a while afterwards, it’s been a long time since I felt a talk of mine failing as I did this time, and it wasn’t until Hanna and some others consoled me by mentioning that they appreciated the contents I did convey (especially the security stuff), and quite a bunch of people found me afterwards with questions, suggestions, and comments.

My slides are available online (bzipped, \~ 1.4 Mib) and I am looking forward to your feedback. I herewith promise never to use OpenOffice.org again.

Saturday night then ended with dinner and a bit of beer drinking, but noone was really up for anything more after the excessive opening party the night before.

Sleep.

Sunday morning, my laptop was still frozen by OpenOffice, and I rebooted it to find myself dumped into busybox as /dev/hda5 (my root) didn’t exist. I quickly found the problem to be a missing udevd and udevsynthesize in the initramfs, but I did not manage to bring the system back up until Tollef gave me a hand to get pivot_root to do what it was supposed to do. Here’s approximately how:

busybox mknod /dev/hda5 b 3 5
mount /dev/hda5 /root
mount -o move /sys /root/sys
mount -o move /proc /root/proc
chroot /root /sbin/udevd --daemon
chroot /root /sbin/udevsynthesize
exec run-init /root /sbin/init 2 < /root/dev/console > /root/dev/console

While we were at it, Daniel Silverstone showed us a wonderful way to help debug problems of this sort: /usr/bin/open. If bash only has /dev/console, it won’t enable job control. This is exactly where open comes in:

/root/usr/bin/open /root/bin/sh

This will put an interactive shell on the next free tty with job control and all. Thanks, Daniel! I think that the above approach actually failed in the end for some weird reason we did not investigate further, so Tollef resorted to fixing the initrd instead:

update-initramfs -c -t -k 2.6.15-1-686

The important part is -t, which essentially redoes the filesystem so that custom changes are undone. I hadn’t made any custom changes, but from the point of view of initramfs, I had deleted the udev binaries.

Within an hour I had a working system again, and a

dpkg-reconfigure linux-image-2.6.15-1-686
zcat /boot/initrd.img-2.6.15-1-686 | cpio -t | grep udev

verified that the problem was only intermittent (I rarely reboot my laptop but update it often).

The T-Shirts Uncle Steve and I were selling pretty much all found new owners. This is great because I had been sitting on about a dozen shirts in XS and XXXL, and I thought I’d never get rid of them. Unfortunately, I sold them at no-profit, but given it was the last batch from Kernel Concepts, I am now thinking of arranging something with Steve to get a bunch of his shirts (which are nicer anyway).

With talks coming to an end that day, everyone’s departure was a stressful time and I was all the more glad to have had another night to spend in the city. I linked up with Jordi and some of the Debian Catalan people who were staying with a friend, Raoul, in the city. Raoul took us around a bit, showed us some of the historic sites in the centre of the city, and finally lead us to the bar of 2000 beers, where supposedly “the best beer of the world” was served: Trappist Westvletern Bleu 8. We ended the night around midnight for our flights were in the early morning.

This having been my first FOSDEM, it was a great experience and I shall definitely try to be in Brussels again next year. Thanks to Wouter and all others who have helped out.

Update: My slide links were previously not working. Thanks to all who pointed that out to me. They are fixed now. See above.

Posted Tue 06 Jan 2009 16:18:48 CET Tags: blog debian ?fosdem ?initramfs ?openoffice ?phd planet-debian

netconf and the GSoC

I am very excited to announce that Jonathan Roes has successfully applied to the Google Summer of Code 2008 and will be working on netconf over this summer. With his help, I am confident that netconf 1.0 in Debian “lenny” is no longer a dream.

Jonathan is a soon-to-be computer science graduate from the University of North Carolina at Charlotte, and his application was (by far) the most convincing of the ten I received. He has several years of programming and Linux experience and it’s quite obvious that he has understood the philosophy of netconf.

I am looking forward to working with Jonathan!

NP: Underworld: Second Toughest of the Infants

Posted Fri 17 Oct 2008 08:25:58 CEST Tags: blog debian ?gsoc ?gsoc2008 netconf planet-debian planet-lca

Netconf status update

If you care about netconf and wonder what’s the current status, you might want to ingest the status update I sent to the netconf-devel mailing list.

The most important point probably is: if you want to work on netconf, I’ll be readily available for questions on the mailing list, even though I cannot expend much time to hack myself over the next four months.

But I will consider coaching anyone who would like to contribute and wants to be coached in doing so. So don’t hesitate and let me know, and consider telling me a bit about what kind of projects you have worked on in the past.

NP: Mono: Under the Pipal Tree

Posted Fri 17 Oct 2008 08:25:58 CEST Tags: blog debian netconf planet-debian planet-lca

Consolidating packaging workflows across distros

I speculate that most of what we do for Debian squares with what others do for their respective distro. Thus, it should be possible to identify a conceptual workflow applicable to all distros, consolidate individual workflows on a per-package basis, and profit from each other. Jonathan let me have the after-afternoon-coffee slot of the Distro Summit for an impromptu discussion on the various workflows used by distros for packaging.

The discussion round was very short-notice and despite the announcement sent to the conference mailing list, only ten people showed up: two people familiar with Fedora, and (“versus”) eight Debianites.

Regardless, I think the discussion was success- and fruitful. We were able to identify a one-to-one mapping between the Fedora and Debian workflows, even though we use different techniques:

• both distros separate original software (“orig tarball”) from modifications made to fit the software in with the rest of the distro.
• Fedora keeps the .spec file, which references the original tarball, alongside any patch files in a per-release directory in their CVS tree, e.g. /mdadm/fedora8 and /glibc/rawhide. To obtain a source tree, the contributor checks out the CVS subtree, downloads the tarball (from their own cache so as to not be at the mercy of upstream) according to the .spec file, and merges the two. There is a tool to automate this, obviously. This process is regularly executed to produce “source RPMs”.
• Debian keeps the original tarball next to a diff.gz file on the mirrors, along with a dsc file which refers to them both. Tools like dget take the URL to the dsc file to download all three, then invoke dpkg-source to unpack the tarball and apply the diff. Individual patch files are either stored in ./debian/patches/ (and applied by the diff), or they don’t exist (meaning that all modifications are concatenated in the diff.gz file.

Many Debian package maintainers use version control systems to maintain the ./debian directory, and if patch files are stored in ./debian/patches/, then Debian and Fedora both store patch files in a version control repository, which seems awful.

Just as I am only one of many who are experimenting with VCS-based workflows for Debian packaging, the Fedora people are also considering the use of version control for packaging. Unlike Fedora, who seem to try to standardise on bzr, I try to cater for the plethora of version control systems in use in Debian, anticipating the impossibility of standardising/converging on a single tool across the entire project.

Update: Toshio Kuratomi wrote in to tell that Fedora has not settled on bzr: “the things that have been tried have spanned most of the current major vcs’s (darcs being the one exception due to it’s not meeting our requirements for keeping history intact.)”

It seems that our two projects are both at the start of a new phase in packaging, a “paradigm shift”. What better time could there be for us to listen to each other and come up with a workflow that works for both projects?

My suggestion currently centres around a common repository for each package across all (participating) distros, and feature branches. Specifically, given an upstream source tree, modifications made during packaging for a given distro fall into four categories:

• upstream changes, such as bug fixes in the original code, or simple things like manpage typos.
• (Linux) distro stuff, such as init.d scripts or Linux-ifications, which upstream doesn’t care about or doesn’t want.
• .deb/.rpm-specific changes, like the ./debian directory or the .spec file.
• distro-specific modifications, like policy compliance and the like.

Given a version control system with sufficient branching support, I imagine having different namespaces for branches: upstream-patches/*, distro/*, rpm/* or debian/*. Now, when building the Debian package, I’d apply upstream-patches/*, distro/*, deb/* and debian/* in order, while my colleague from the Fedora project would apply upstream-patches/*, distro/*, rpm/* and fedora/*, before calling the build tools and uploading the package.

There are surely problems to be overcome. Pascal Hakim mentioned patch dependencies, and I can’t necessarily say with a clear conscience that my workflow isn’t too complicated to be unleashed into the public… yet. But if we find a conceptual workflow applicable to more than one distro, it should be possible to implement a higher-level tool to implement it.

Also, the above is basically patch maintenance, not the entire workflow. Bug tracking system integration is going to play a role, as well as other aspects of daily distro packaging. I’ll leave those for future time.

For me, this is the start of a potentially fruitful cooperation and I hope that interested parties from other distros jump on. For now, I suggest my mailing list for discussion. You can also find some links on the Debian wiki.

Posted Fri 17 Oct 2008 08:25:58 CEST Tags: blog debian ?fedora planet-debian planet-lca ?vcs vcs-pkg ?workflow

RRR Interview on Debian and cross-distro collaboration

During LCA2008, Ed Borland of Melbourne-based Triple R FM Byte Into It show took me aside for an interview and asked some good questions about Debian and my work on cross-distro collaboration. The interview was recorded and is now available as Ogg Vorbis file from the 14 May 2008 issue of Byte Into It.

I am looking forward to any feedback.

Thanks to Ed and Phil Wales for their time and help.

NP: Mono: You Are There

Posted Fri 17 Oct 2008 08:25:58 CEST Tags: blog ?cross-distro debian ?interview planet-debian planet-lca vcs-pkg

The state of the Debian project

The first of my talks at LCA 2008 gave me a chance to talk about the current state of the Debian project, which got me my first LWN.net coverage — with a photo, even (subscribers only for now, the article will become public on 7 February). Thank you, Jonathan Corbet, for a very good article which nailed all the main points!

Slides are here.

I agreed to this talk on short notice because I like to talk about Debian and was honoured by the chance to represent the project in this form. I would not have been able to do it without plenty of helpful input from colleagues in the last few days. Since I didn’t get a chance to display the final slide with the acknowledgements during the talk, I would herewith like to thank specifically Andreas Tille, Michael Banck, Kevin Mark, Josip Rodin, MJ Ray, Cyril Brulebois, Stefano Zacchiroli, Frans Pop, Moritz Mühlenhoff, Russ Allbery, Steve Langasek, Luk Class, Andreas Schuldei, and Christian Perrier. No guarantees for the completeness of the list.

The talk ended in an open discussion on how Debian could improve. I took notes and shall forward them to the project mailing list, once I get a chance. Thanks to all the participants, as well.

Update: during the talk, I mentioned that there was no security support around the time of the “etch” release. Thanks to Moritz Mühlenhoff, who spotted my error: that should have been “sarge”. The problems with security support had long been resolved by the time “etch” was being prepared, and this was in (large?) part thanks to Moritz. Sorry for the screwup!

I also called Linux to tend towards multimedia more than one might like. I should not have made this comment as a representative of the Debian project, and I probably did unjust to the Linux kernel in whole. This is entirely a personal issue, I have a number of problems with Linux memory management, scheduling, and some other points relevant to production use. I’ve had some of these problems for years, but they seem never to get fixed, while development is fast-paced. Then I look at some of the work being done and I wonder what the priorities are.

Regardless, I should not have made this comment and I apologise for it.

Update: bzip2-compressed source packages are not yet supported. Thanks, Lior, for alerting me to this.

Update: the fourth issue of the misc development news mentiona a few other topics which I forgot, such as the Debian Enhancement Proposals.

Posted Fri 17 Oct 2008 08:25:58 CEST Tags: blog debian planet-debian planet-lca