home of the madduck/ blog/
Up the Khyber!

What started with a little host naming theme riddle, and spread as a meme across various planets, has come full circle today, as I installed a new machine, khyber.madduck.net in the racks hosted at the LayerOne colocation site in Zürich.

Init Seven, my favourite Swiss ISP, are sponsoring the rack space and connectivity: a routed /28 IPv4 subnet and a /64 IPv6 subnet.

The machine is a Transtec Calleo 331 1U server with two quad-core AMD Opteron 2354 processors, 16Gb of RAM, and 4Tb of disk space, configured into a 2Tb RAID1 with LVM on top. It will be used to host a couple of KVM instances for a number of projects I currently operate off my main, personal server, pulse.madduck.net.

Unfortunately, Transtec (as well as most other vendors I checked out) uses nVidia's MCP55Pro chipset, which had me sceptical long before I sent the order. nVidia is known to suck at thinking open-source, and the Linux kernel driver for their network chips, forcedeth, had to be reverse-engineered, because nVidia failed to cooperate. Apart from that, nVidia's chips are also often simply crap. Many of their motherboards come with broken APICs and the network chips often wreak havoc on subnets.

I find it curious that a search for "MCP55" or "MCP55Pro" on the nVidia website yields no results.

Given these expectations, I insisted on a try-and-buy contract, which are not default in Switzerland (as opposed to the EU). My fears proved right, it wasn't long until I encountered the first problems with nVidia's crap: the network chip is unable to deal with segmentation offloading for larger packets, e.g. those with IPv6 payload in a bridging context, and generic segmentation offloading has to be turned off on the interface:

ethtool -K eth0 gso off

This fixes the issue, and so I stuffed it into /etc/network/interfaces and decided to keep the machine, especially after having invested the time into the server to figure out how to work the watchdog and the IPMI card. The folks behind Transtec's support hotline are helpful, but it took me a bit of time to get the information I wanted out of them. At least they deemed it perfectly natural that I wanted to run Debian, and never questioned that.

Let's document some of the more internal specs here for posterity:

The BIOS gives the impression of two separate watchdogs, one on the IPMI card (a SIMSO+) and one on the mainboard itself (a SuperMicro H8DMU+). Transtec told me that SuperMicro disabled the IPMI watchdog (called "BMC watchdog" in the BIOS) with firmware 1.29, leaving only the one on the motherboard. It took a bit of time to find out that this was a W83627THF and supported by the w83627hf_wdt driver, which I configured with module options timeout=120 nowayout=1. I left the BIOS setting at "disabled" since that would otherwise interfere with booting e.g. the awesome grml live system by rebooting the machine after the set amount of time.

It also took a bit of time to get SOL working with the IPMI card. In the end, I found it to work best by configuring the console redirection in the BIOS to use COM2 with 38400/n/8, using hardware flow control, redirection-after-POST always, and vt100 emulation — ANSI and vt-utf8 emulation were horrific. I instructed the bootloader to pass console=tty0 console=ttyS1,38400n8 to the kernel (order matters), and configured agetty in /etc/inittab like this:

T0:2345:respawn:/sbin/getty -8hL ttyS1 38400n8 vt100

To guard against the worst, I also left a grml CD in the drive, to be able to boot a rescue system via SOL in case everything else was broken. The following kernel boot parameters enable me to use the rescue system via IPMI:

vga=normal noeject noprompt console=tty0 console=ttyS1,38400n8 forensic

Michael Prokop of the grml project tried to create a customised ISO image with these options for me, but I kept getting medium errors trying to read it, independent of the burner, CD-writing-software, operating system used to burn, medium, and drive used to read, so it's possible that the ISO image is faulty. Michael used the following command line to create it (supporting files are here):

grml-live -t ~/templates-madduck -c GRMLBASE,GRML_MEDIUM,AMD64 -o \
  /grml-live/grml64-medium_2008.11-madduck -a amd64 -s sid -v \
  2008.11-madduck -g grml64-medium -r Schluchtenscheisser -V -b

Until we figure out the problem with this, the grml64 0.2 CD in the drive will do.

The machine is now operational, but the IPv6 prefix I have still arrives at vera.madduck.net, a Xen-Instance hosted by Init Seven, which I will no longer need, once I find the time to migrate all the services it hosts.

NP: This Will Destroy You: Young Mountain