Linux.conf.au 2010 has come to an end and I am looking back at an intense week of conferencing. A big shout out to the organisers for their excellent work. I think LCA (as well as DebConf) just keeps getting better every year. This does not at all discredit previous organisers, because they were the best at their times and then passed on the wisdom and experience to help make it even better in the following year.
The week started off with the DistroSummit, which Fabio and I organised. Slides are forthcoming, as I failed to get them off the speakers right after their talks — it's interesting how stress levels and adrenaline can cause one to forget the most obvious things. This is where experience comes in. I'll be there again next year, I hope, to do things better.
The theme of the day was cross-distro collaboration, and we started the day a little bit on the Debian-side with Lucas Nussbaum telling us about quality assurance in Debian, alongside an overview of available resources. We hoped to give people from other distros pointers, and solicit feedback that would enable us to tie quality assurance closer together.
Next up was Bdale Garbee who talked about the status of the Linux Standard Base. While I am really interested in such standardisation efforts, I realised during his talks that I had considerable difficulties paying attention because as organiser of the conference, I had all sorts of other things occupying my thoughts.
I proceeded to tell the audience — the room was mostly filled throughout the day with an estimated 40–50 folks, and I'd say about half of them stayed throughout, while the other half came in and left the room between talks. I could not get the projector to work with my laptop after the upgrade to Kernel Mode Setting, and thus used the whiteboard to give a brief introduction to vcs-pkg.org, talk about the current state of affairs, summarise the trends in discussions around patch management and collaboration, give an outlook of what's up next, and solicit some discussion.
Sadly, just like during Bdale's talk, I found myself worrying over the organisation of the day, rather than actually taking in most of the discussion. Fortunately, others have written about the most important points, so I defer to them.
Michael Homer then told us about GoboLinux's Aliens system, which is a way to integrate domain-specific packages with distro-specific package maintenance — e.g. how to get APT to handle CPAN directly, or how to let YUM manage Python packages. The ensuing discussion was interesting, and we carried it over to the next slot, because Scott, the next speaker, was stuck in traffic. To summarise briefly: scripting languages have a lot of NIH-style solutions because it works for them, but these are a nightmare to distro packagers. One symptom of the status quo is that complex software packages like Zimbra are forced to distribute all required components in their installation packages, which make distro packaging, quality assurance, and security support even harder. I don't think we found a solution, other than the need for further standardisation (like the LSB), but the road seems to be a long and windy one.
Laszlo Peter introduced the audience to SourceJuicer, a new build system used by OpenSolaris. The idea is that contributors submit packages via a web interface, kicking off a workflow incorporating discussion and vetting, and only after changes have been signed-off are packages forwarded to auto-builders and eventually end up in the package repository. This is very similar to upload ideas I've had a while ago, which I've started to (finally) implement. Unfortunately, SourceJuicer seems very specific to OpenSolaris, as well as non-modular, so that I probably won't be able to reuse e.g. the web interface on top of a Debian-specific package builder.
After the break, Dustin Kirkland stepped up to tell us about his user experience of Launchpad. Unfortunately, I found his talk a bit too enthusiastic. Launchpad undoubtedly has some very cool features and ideas, but it's just one of the available solutions.
The dicussion of Launchpad also dominated the next talk, in which Lucas Nussbaum told us about the Debian-Ubuntu relationship. While his presentation showed that the relationship was improving (Matt Zimmerman made the point that there are rather many relationships, rather than one relationship), I was a bit disturbed by the comments of Launchpad developers in the room, ranging from "Debian is declining anyway" to "Just use Launchpad if you want to collaborate with others and not go down". There was a slight aura of arrogance in their comments which tainted my experience of the otherwise constructive discussions of the day.
Overall I had a great time. Debian and Ubuntu made up the vast majority of attendants, with only a handful of representatives from other distros present. I wonder why that would be. One reason might be that around 70% of LCA attendants declared themselves Debian or Ubuntu users, and so there weren't many other distros around. Another might be that I still haven't spread the word enough. Let's hope to do better next year!
Thanks to all the speakers. We may have organised the day, but you made it happen and interesting!
Slides and recordings of the talks will be linked from the archived website when they become available (yes, the archive page does not exist yet either).
Update: Jelmer informed me that the people who spoke up against Debian during and after the Launchpad talk were not officially affiliated with Launchpad. It's a shame that this negatively reflected upon Launchpad for some of the attendees (not just myself).
It took us a bit longer than planned, but we are happy to announce the schedule for the Distro Summit at the upcoming LCA2010 conference. We focused on cross-distro aspects, and we hope that you are as excited as we are about the result.
The schedule is displayed on the Distro Summit homepage and I best refrain from duplicating it here.
Today, Fabio and I extended the deadline of the call for papers of the upcoming Distro Summit until 18 October 2009. We have already received a number of great proposals, but not enough to complete the schedule. So if you have anything to say on the topic of distributions (such as, but not limited to the various Linux distributions), and especially on cross-distro collaboration, please launch your mailer and compose a message as detailed in the instructions.
Distro Summit will take place as part of LCA 2010 in Wellington, New Zealand. Yes, that is on the Southern Hemisphere, and yes, that is far away from where many of you live, but it's well worth a visit, if you'll take my word for it. Unfortunately, we are unable to provide sponsorship for speakers, so talk to your employer or your distro now and get the ball rolling. You have a little more than a week to get your bids in!
What are you waiting for? This is a once-in-a-lifetime chance: Distro Summit, LCA, and Wellington, all-in-one. It doesn't get much better than that. Let's hear your proposal!
PS: I loved Wellington (and its people) so much that I picked up my very own pet Wellingtonian!
The other day, Romain shared his concerns about using Git for team-maintained packaging. His comment system is currently broken, so I wrote an e-mail reply, which I would like to share.
I agree with Romain that the design decision to not support subtree checkouts like SVN is not without problems. As opposed to a single SVN repo with components in subdirectories that you can individually check out, you might end up with a hundred Git repos, and the same change to all then requires one to iterate all 100.
I'd like to make the distinction between trivial changes (e.g.
s/© 2008/&-2009/g) and those that might not be
Standards-Version, or something even more
In case of the former, there's no question, it can be painful to operate across a hundred repos. Tools like mr make that a bit easier, but it's still far from optimal.
The latter, however — updating
and adding the appropriate changelog entry — is not really
comparable. Neither would be e.g. changing a file location in 100
different repos. In those cases, every single package needs manual
intervention, and if only for quality reasons and testing. In this
sense, I actually think that a single SVN checkout with all the
subtrees and the possibility to easily commit the result of a
recursive action is counter-productive.
On the other hand, I don't say that I am pleased with the workflow Git (or any other DVCS for that matter) imposes. It's sometimes quite painful, as Romain says. We are missing higher-level tools that allow for easier and more intuitive bulk operations. I think that they should be implemented outside of the VCS-tool though, true to the Unix principles. SVN integrates it all into a monolithic piece of software, and that often isn't ideal either (think size and slowness, or backup weight, or chance of corruption, or granular access control, or the impossibility to properly track files across subtrees).
mr is a step in the right direction, and we need
more tools along those lines. First, however, I think that people
need to figure out how exactly to use DVCS for packaging, such that
there is any chance of consolidating workflows across a larger
number of packages; if everyone does it their own, slightly unique
way, then that goal is inifinitely far. This is the reason I
started vcs-pkg.org, and even
though we're still far from anywhere, I am quite pleased with what
we've done so far.
If you're at Debcamp or DebConf, maybe you could join the discussion.
Romain also mentioned that distributed VCS don't allow for the same sort of centralisation as SVN does. I disagree: you can use Git in exactly that way, as a centralised repo from which packages are built. The nice advantage over SVN (one which svk tried to close) is the ability for everyone to easily branch/fork, or work offline.
Once you start down that path, it somehow inherently becomes everyone's own responsibility to ensure that one's changes end up in the central repository (where commit hooks might verify the build-ability, ensure that the test suite still passes, or run simple format/consistency checks).
This sort of workflow is very different from the one with a self-appointed benevolent dictator at the top, who (like Linus, or Junio for Git) sometimes forget to include patches due to overflooding. The question is really: Given that you need some sort of centralised release coordination, do you want a human or a repo to be the central entity (and single point of failure)?
I really prefer the repo, since that places the sole responsibility on the leafs, on the contributors, who need to see their code through all the way.
It's a whole lot more rewarding to commit/push, get a rejection, pull, merge, commit/push, and be done, rather than to send a patch to upstream, wait, reping, notice that it's not in in the new release, ask, ping, change, reping, get angry, ping, hope, wait, ping, wonder why the heck you are still doing this, write angry email but don't send it, reping, ask, and finally notice that it's been accepted after all.
NP: Deep Purple: Made in Japan
Manoj (thanks to Jaldhar, I finally know how to pronounce your name... it would be spelt "Manosch" in German and bears resemblance to the name of one of the greater German child book authors)... uh, so where was I?
Oh yeah, Manoj has a really interesting way to manage packages with GNU arch, which I have started to adopt for my packages. The approach gets rid of things like dpatch (yes!) and simply manages additional features and debianisation in separate branches. Read the last paragraph of this entry if you can't be bothered with the rest.
Even though bazaar still has its rough edges, it's a pleasure to work with the guys making it possible, and bugs have quick turnaround times. This is one of the reasons why I chose to adopt Manoj's method together with bazaar for pkg-zope as well as pkg-mdadm. I have started to document the process on the pkg-zope wiki.
The only real drawback of Manoj's method is that it has a steep learning curve. Thus, Mario (the other mdadm maintainer) and I had the idea to hold a live session on IRC in which to demonstrate the use of arch and allow people to ask questions.
We are meeting for this live demonstration and Q&A session
tomorrow, Thursday 11 August 2005, at 19:00 hours GMT in the
#pkg-zope channel on irc.debian.org, and you are
cordially invited to join. I'll also log the session and make it
available online afterwards. Hope to see you there.
I speculate that most of what we do for Debian squares with what others do for their respective distro. Thus, it should be possible to identify a conceptual workflow applicable to all distros, consolidate individual workflows on a per-package basis, and profit from each other. Jonathan let me have the after-afternoon-coffee slot of the Distro Summit for an impromptu discussion on the various workflows used by distros for packaging.
The discussion round was very short-notice and despite the announcement sent to the conference mailing list, only ten people showed up: two people familiar with Fedora, and ("versus") eight Debianites.
Regardless, I think the discussion was success- and fruitful. We were able to identify a one-to-one mapping between the Fedora and Debian workflows, even though we use different techniques:
- both distros separate original software ("orig tarball") from modifications made to fit the software in with the rest of the distro.
- Fedora keeps the
.specfile, which references the original tarball, alongside any patch files in a per-release directory in their CVS tree, e.g.
/glibc/rawhide. To obtain a source tree, the contributor checks out the CVS subtree, downloads the tarball (from their own cache so as to not be at the mercy of upstream) according to the
.specfile, and merges the two. There is a tool to automate this, obviously. This process is regularly executed to produce "source RPMs".
- Debian keeps the original tarball next to a
diff.gzfile on the mirrors, along with a
dscfile which refers to them both. Tools like
dgettake the URL to the
dscfile to download all three, then invoke
dpkg-sourceto unpack the tarball and apply the diff. Individual patch files are either stored in
./debian/patches/(and applied by the diff), or they don't exist (meaning that all modifications are concatenated in the
Many Debian package maintainers use version control systems to
./debian directory, and if patch files
are stored in
./debian/patches/, then Debian and
Fedora both store patch files in a version control repository,
which seems awful.
Just as I am only one of many who are experimenting with VCS-based workflows for Debian packaging, the Fedora people are also considering the use of version control for packaging. Unlike Fedora, who seem to try to standardise on bzr, I try to cater for the plethora of version control systems in use in Debian, anticipating the impossibility of standardising/converging on a single tool across the entire project.
Update: Toshio Kuratomi wrote in to tell that Fedora has not settled on bzr: "the things that have been tried have spanned most of the current major vcs's (darcs being the one exception due to it's not meeting our requirements for keeping history intact.)"
It seems that our two projects are both at the start of a new phase in packaging, a "paradigm shift". What better time could there be for us to listen to each other and come up with a workflow that works for both projects?
My suggestion currently centres around a common repository for each package across all (participating) distros, and feature branches. Specifically, given an upstream source tree, modifications made during packaging for a given distro fall into four categories:
- upstream changes, such as bug fixes in the original code, or simple things like manpage typos.
- (Linux) distro stuff, such as
init.dscripts or Linux-ifications, which upstream doesn't care about or doesn't want.
- .deb/.rpm-specific changes, like the
./debiandirectory or the
- distro-specific modifications, like policy compliance and the like.
Given a version control system with sufficient branching
support, I imagine having different namespaces for branches:
debian/*. Now, when building the
Debian package, I'd apply
in order, while my colleague from the Fedora project would apply
fedora/*, before calling the
build tools and uploading the package.
There are surely problems to be overcome. Pascal Hakim mentioned patch dependencies, and I can't necessarily say with a clear conscience that my workflow isn't too complicated to be unleashed into the public… yet. But if we find a conceptual workflow applicable to more than one distro, it should be possible to implement a higher-level tool to implement it.
Also, the above is basically patch maintenance, not the entire workflow. Bug tracking system integration is going to play a role, as well as other aspects of daily distro packaging. I'll leave those for future time.
For me, this is the start of a potentially fruitful cooperation and I hope that interested parties from other distros jump on. For now, I suggest my mailing list for discussion. You can also find some links on the Debian wiki.
During LCA2008, Ed Borland of Melbourne-based Triple R FM Byte Into It show took me aside for an interview and asked some good questions about Debian and my work on cross-distro collaboration. The interview was recorded and is now available as Ogg Vorbis file from the 14 May 2008 issue of Byte Into It.
I am looking forward to any feedback.
Thanks to Ed and Phil Wales for their time and help.
NP: Mono: You Are There
Currently, the tentatively scheduled dates are 2-7 September 2008. You can get the details from the wiki page. If you're interested, please reserve those dates and add yourself to the list of participants.
NP: Hooverphonic: The Magnificent Tree
If you are interested in using version control for distro packaging, you
- might like http://vcs-pkg.org.
- might want to join
- should be signed up to our mailing list.
If you read the mailing list, you know about the upcoming Extremadura meeting 2-6 April 2008.
If this is news to you, well, it isn't anymore.
If you think you should be in Extremadura when this party takes
place, don't hesitate and reply. The message ID is
Update: mostly due to the short notice, I had to call off the meeting. I will make a run for the next slot and hopefully announce it a lot earlier.