Yesterday's RAM mystery was solved after dozens of people have written in with helpful comments. Thank you all. I need to find time to create a proper, commentable blog soon so that you all can get the credit you deserve!
I now have the full 4GB of RAM available:
$ free | head -2 total used free shared buffers cached Mem: 4063324 1657440 2405884 0 149080 643732
The suggestions ranged from timing issues to defective modules, from 64bit OS via south/north bridge limitations to sacrificing chickens to god spongebobs.
In the end, several people put their finger on a concept called "memory hole remapping", and that turned out to be the ticket. Here is what I learnt:
This motherboard and the CPU memory controller supports (which means "can address") 4096MB of RAM. Since extensions cards, such as the graphics card and the network controller, also need addressable slabs of memory for their RAM or registers, ranges in the upper memory regions are mapped to the RAM of those cards.
/proc/mtrr gives a clue over what's
reg00: base=0x000000000 ( 0MB), size= 2048MB, count=1: write-back reg01: base=0x080000000 ( 2048MB), size= 1024MB, count=1: write-back reg02: base=0x0c0000000 ( 3072MB), size= 256MB, count=2: write-combining reg03: base=0x0d8000000 ( 3456MB), size= 128MB, count=1: write-combining reg04: base=0x0e8000000 ( 3712MB), size= 64MB, count=1: write-combining reg05: base=0x0ec000000 ( 3776MB), size= 64MB, count=1: write-combining
Registers 0 and 1 represent the addressable RAM available to the operating system. At position 3072, we find two slabs of 256MB each, most likely corresponding to the two graphics cards in the system. I think that the 128MB in register 3 is the AGP aperture. Registers 4 and 5 correspond to 64MB slabs each, most likely relating to the two network cards I have installed.
The result is that addresses in memory beyond 3072MB are "routed" to the extension cards, which "overshadow" the original system RAM and leave it unusable.
When I turned on the "memory hole remapping" feature in the BIOS, I empowered the BIOS to remap the memory space. The way it does this is BIOS-dependent, but most likely, it will remap the overshadowed memory region to the end of the total memory space. This is a software-only remapping, but the effect is that the system appears to have more memory.
/proc/mtrr now reads:
reg00: base=0x000000000 ( 0MB), size= 4096MB, count=1: write-back reg01: base=0x100000000 ( 4096MB), size= 1024MB, count=1: write-back reg02: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable reg03: base=0x0c0000000 ( 3072MB), size= 256MB, count=1: write-combining
I am not entirely sure how to interpret this yet, but you can see that there are now an additional 1024MB at position 4096 (register 1), so the system seems to have 5GB of RAM available. Register 2 describes memory between 3072 and 4096 as not cacheable, because it maps to extension cards. I do not know how register 3 fits into the picture.
By remapping, we cross the 4096MB mark. To make use of the additional memory, we will need PAE enabled to give you 36 bits, or a 64bit operating system: 2^32 is 4096M, so with only 32 bits available, 4GB is all the memory that can ever be addressed (dirty hacks excluded). Fortunately, this system has been running the Debian amd64 port even before it was officially released, and thus, once the BIOS remapped the memory, I could fully make use of it.
Mystery solved, some knowledge acquired, many of you left to thank!