SME Server 6.0.1 on DL 380 G4 How-To

Author : Pierluigi Miranda
Sources : original work
Tested on : SME Server Version 6.0.1 GPL
Updated : 17 Feb 2005
License : GPL

Abstract : the onboard DL 380 G4 Smart Array 6i raid controller is not supported by the SME Server 6.0.1 kernels. This document explains in a certain detail the procedure to update the cciss.o kernel modules to the latest version available and install SME Server on these machines.

Even if this procedure is targeted to a specific hardware, it can be adapted to update the drivers of a generic block device supported by the SME Server system, and modified to include support for new block devices, provided that they are available as kernel module sources for the 2.4.x kernel family.


The HP Proliant DL 380 G4 onboard Smart Array 6i raid controller is not recognized by the 2.4.20-18.7 kernels used by SME Server - the controller is younger than them. The Smart Array family cciss.o drivers are compiled as kernel modules, so it should be viable to update them without compiling the whole kernel.

What are we going to do?

Basically, we need to do a SME Server clean installation, add kernel and driver sources and development packages, compile the modules needed and figure out a way to insert them in the Proliant system during the installation process.

I started configuring a SME Server as Server only on a clean i686 PC (a VMware virtual machine, to be nitpicky), then I installed the kernel source, the development tools and the cciss driver sources (released as an Open Source Sourceforge project).
On this system I compiled the cciss.o kernel modules for each kernel involved (the BOOT, UP and SMP ones), updated the drvblock.img additional block device driver disc available in the /images/ SME Server CDROM with my new BOOT kernel module, modified the boot.img installation floppy (why this step is needed will be clear later) and created a floppy with the UP and SMP cciss.o modules.

Thus I was able to force the installer to ask for additional drivers, load my new cciss.o driver, install SME Server as usual then shell to a command line before rebooting to copy the updated driver on the system and do a bit of reconfiguration as needed.


The quick and dirty way

Thanks to the Contribs.org community, the modified disc images created in the following procedure are now available in my contribs area; so, you can download them and start installing, sparing yourself the tedious work of compiling kernel modules and updating floppy images. That's what you need:

Create the disk from the images using the dd if=filename of=/dev/fd0 bs=1440k command and you can skip right to the Installation time - showtime at last! section.


Some preparatory workout

To begin, logon as root on the console and create on the new SME Server (the VMware machine) a "sandbox" directory that will contain all our work, and the subdirectory to put the kernel source and developer packages into:

# mkdir -p /SME-6.0.1-01-cciss/rpm-devel

Download all the available packages from Contribs.org in this directory, then install them - there are some dependency problems, so we must force the installation - no problem anyway in our case:

# cd /SME-6.0.1-01-cciss/rpm-devel
# rpm -Uvh --nodeps *.rpm

Time to get the cciss.o driver sources from the SourceForge CCISS Project: the driver are available in .tar.gz format, and we need the 2.4.50 version (newer versions are for 2.6.x kernels). Create a directory for the cciss.o sources:

# mkdir -p /SME-6.0.1-01-cciss/cciss

and download the archive in it.

Now, the cciss.o sources must be inserted in the kernel source tree, substituting the old ones; let's do it (but keep a copy of the old drivers and kernel Makefile, just for good practice):

# cd /usr/src/linux-2.4/drivers/block
# mv cciss_cmd.h cciss_cmd.h.original
# mv cciss.c cciss.c.original
# mv cciss.h cciss.h.original
# mv cciss_ioctl.h ccis_ioctl.h.original
# mv cciss_scsi.c cciss_scsi.c.original
# mv cciss_scsi.h cciss_scsi.h.original

# cd /usr/src/linux-2.4/include/linux
# mv cciss_ioctl.h cciss_ioctl.h.original

# cd /usr/src/linux-2.4/Documentation
# mv cciss.txt cciss.txt.original
# mv mkdev.cciss mkdev.cciss.original

# cd /usr/src/linux-2.4
# tar xvzf /SME-6.0.1-01-cciss/cciss/cciss_2.4.50.tar.gz

# cp -a Makefile Makefile.original


Compilation time - the fun starts here

As we know well, the cciss.o kernel modules must be compiled three times, one for each kernel involved (2.4.20-18-7BOOT, 2.4.20-18-7BOOT, 2.4.20-18-7smp), each time using the right configuration file and telling make what kernel the modules are to be compiled for.

The configuration file issue is a matter of copying the right file in the right place with the right name, while the kernel version and type must be specified editing the Makefile and setting the EXTRAVERSION variable accordingly.

# cd /usr/src/linux-2.4
# cp -a Makefile.original Makefile
cp: overwrite `Makefile'? y
# make clean mrproper
# cp configs/kernel-2.4.20-i386-BOOT.config .config
# vi Makefile

(EXTRAVERSION = -18.7BOOT)

# make oldconfig

(or make menuconfig, then immediately Exit and save configuration when asked)

# make dep
# make SUBDIRS=drivers/block modules

Store the compiled modules away:

# mkdir -p /SME-6.0.1-01-cciss/lib/modules/2.4.20-18.7BOOT/kernel/drivers/block
# cp drivers/block/cciss.o /SME-6.0.1-01-cciss/lib/modules/2.4.20-18.7BOOT/kernel/drivers/block/

# cd /usr/src/linux-2.4
# cp -a Makefile.original Makefile
cp: overwrite `Makefile'? y
# make clean mrproper
# cp /boot/config-2.4.20-18.7 .config
# vi Makefile

(EXTRAVERSION = -18.7)

# make oldconfig

(or make menuconfig, then immediately Exit and save configuration when asked)

# make dep
# make SUBDIRS=drivers/block modules

Store the compiled modules away:

# mkdir -p /SME-6.0.1-01-cciss/lib/modules/2.4.20-18.7/kernel/drivers/block
# cp drivers/block/cciss.o /SME-6.0.1-01-cciss/lib/modules/2.4.20-18.7/kernel/drivers/block/

# cd /usr/src/linux-2.4
# cp -a Makefile.original Makefile
cp: overwrite `Makefile'? y
# make clean mrproper
# cp /boot/config-2.4.20-18.7smp .config
# vi Makefile

(EXTRAVERSION = -18.7smp)

# make oldconfig

(or make menuconfig, then immediately Exit and save configuration when asked)

# make dep
# make SUBDIRS=drivers/block modules

Store the compiled modules away:

# mkdir -p /SME-6.0.1-01-cciss/lib/modules/2.4.20-18.7smp/kernel/drivers/block
# cp drivers/block/cciss.o /SME-6.0.1-01-cciss/lib/modules/2.4.20-18.7smp/kernel/drivers/block/


Let's juggle floppy discs

The installation CD-ROM has a flaw: even if you boot keeping the SHIFT key pressed and type accept dd at the boot: prompt, you are not allowed to select an additional device driver to load.

The installer asks if you want to load additional devices and even lets you insert the floppy in the drive, but then ignores it and proceed with a standard installation until it cannot find a hard disk drive to write data on.

The installation must then be started from the boot floppy.
But the boot floppy is flawed, too: the kickstart configuration file is missing, so the boot.img image must be modified.

As usual, let's create a directory for floppy images manipulation:

# cd /SME-6.0.1-01-cciss
# mkdir -p floppy/files
# cd floppy

Insert the SME Server CDROM in the driver and copy the boot.img image

# mount /mnt/cdrom
# cp /mnt/cdrom/images/boot.img .

then add the missing kickstart configuration file

# cp boot.img boot.img.original
# mount -o loop -t auto boot.img /mnt/floppy
# cp /mnt/cdrom/images/ks_stub.cfg /mnt/floppy/ks.cfg
# umount /mnt/floppy
# umount /mnt/cdrom

The real floppy disk needed to start the installation can be created from the image with the command:

dd if=boot.img of=/dev/fd0 bs=1440k

The additional block device driver disc drvblock.img from the /images/ directory of the SME Server CDROM can now be updated with the new cciss.o boot kernel module needed at installation time.

Copy the floppy image from CDROM to hard disk

# mount /dev/cdrom /mnt/cdrom
# cp /mnt/cdrom/images/drvblock.img .

then transfer its content to hard disk, too:

# mount -o loop -t auto drvblock.img /mnt/floppy
# cp /mnt/floppy/* files/
# umount /mnt/floppy

The old cciss.o kernel module is stored in the cpio gzipped archive modules.cgz and must be substituted with the new one we compiled before. So we have to decompress the archive, extract the files stored in it, overwrite the cciss.o module and recreate the compresse archive in the format required by the installer:

# cd files
# mv modules.cgz modules.cgz.original
# zcat modules.cgz.original | cpio -iv --make-directories
# cp /SME-6.0.1-01-cciss/lib/modules/2.4.20-18.7BOOT/kernel/drivers/block/cciss.o 2.4.20-18.7BOOT/
# find 2.4.20-18.7BOOT | tail +2 > cpio.list
# cpio -o -H crc < cpio.list | gzip -c9 - > modules.cgz

The pcitable file must be updated, too. So create a patch file diff.patch containing the text that follows:

 3,4c3,11 
 < 0x0e11 0xb060 "cciss" "Compaq   Smart Array 5300 Controller" 
 < 0x0e11 0xb178 "cciss" "Compaq   Smart Array 5i/532" 
 --- 
 > 0x0e11 0x4070 "cciss" "Compaq   Smart Array 5300" 
 > 0x0e11 0x4080 "cciss" "Compaq   Smart Array 5i" 
 > 0x0e11 0x4082 "cciss" "Compaq   Smart Array 532" 
 > 0x0e11 0x4083 "cciss" "Compaq   Smart Array 5312" 
 > 0x0e11 0x409a "cciss" "Compaq   Smart Array 641" 
 > 0x0e11 0x409b "cciss" "Compaq   Smart Array 642" 
 > 0x0e11 0x409c "cciss" "Compaq   Smart Array 6400" 
 > 0x0e11 0x409d "cciss" "Compaq   Smart Array 6400 EM" 
 > 0x0e11 0x4091 "cciss" "Compaq   Smart Array 6i" 


Pay attention that the four fields on each line be separated by a single TAB - don't know if it's required, but it's the original file format.

Then patch the pcitable file:

# cp pcitable pcitable.original
# patch pcitable diff.patch

Now we are ready to create a new updated additional block device driver disc image drvblock.img:

# cd ..
# cp drvblock.img drvblock.img.original
# chmod a+rw drvblock.img
# mount -o loop -t auto drvblock.img /mnt/floppy
# cp files/modules.cgz files/pcitable /mnt/floppy
cp: overwrite `/mnt/floppy/modules.cgz'? y
cp: overwrite `/mnt/floppy/pcitable'? y
# umount /mnt/floppy

The real floppy disk needed during the installation can be created from the image with the command:

dd if=drvblock.img of=/dev/fd0 bs=1440k

It's now time to create an image containing the cciss.o UP and SMP kernel modules that will be used by the installed system.

First, archive the kernel modules in a single file:

# cd /SME-6.0.1-01-cciss
# tar cvzf floppy/cciss.o.tgz lib/modules/2.4.20-18.7/kernel/drivers/block/cciss.o \
lib/modules/2.4.20-18.7smp/kernel/drivers/block/cciss.o
# cd floppy

Then create a blank floppy image and format it:

# dd if=/dev/zero of=cciss.o.img bs=1440k count=1
1+0 records in
1+0 records out
# losetup /dev/loop7 cciss.o.img
# mke2fs -c /dev/loop7
mke2fs 1.27 (4-Mar-2002)
(...)
Writing superblocks and filesystem accounting information: done
(...)
# losetup -d /dev/loop7

Now copy the gzipped tar on the floppy image:

# mount -o loop -t auto cciss.o.img /mnt/floppy
# cp cciss.o.tgz /mnt/floppy/
# umount /mnt/floppy

The real floppy disk needed at the end of the installation can be created from the image with the command:

dd if=cciss.o.img of=/dev/fd0 bs=1440k


Installation time - showtime at last!

Ingredients:
  • the modified boot disc
  • the updated additional block device driver disc
  • the cciss.o kernel modules disc
  • the SME Server CD-ROM, the HP Proliant DL 380 G4 - of course :-)

We have to trick the installer to force it to load our updated cciss.o driver: so boot from the floppy, leaving the CDROM reader empty.

The installer will not find the SME server CDROM and will ask for it: press Back, and the installer will proceed asking you the language of choice for installation and the keyboard type of the system. Then it will ask you which media contains the packages to install.

Select the (non existent) hard drive: the installer will reply that no hard drives are available, and will offer you to configure additional devices. Press Yes, then F2 to load an additional driver disk.

Put our additional block device driver disc in the floppy drive and press OK, then choose Compaq Smart Array 5xxx Controller (cciss.o): the cciss.o driver will be loaded, and the Smart Array 6i recognized by the system.

If some other version of linux has already been installed on this system, the installer will ask what partition contains the SME Server CD image; press Back; otherwise, you should already be back to the install packages media selection: now, at last, insert the SME Server CDROM and select Local CDROM.

The EULA will be displayed, and the standard installation will proceed as usual - of course, if the volumes served by the Smart Array 6i are already formatted, the installer will ask your confirmation to destroy their content.

Proceed with the standard installation until its completion, but when presented with the Complete notice DO NOT press OK to reboot, if you do not want to attend to a kernel panic and restart the installation again: press CTRL-ALT-F2 instead, and take full control of the system in a console command prompt.

Free the floppy device (the driver disc is still mounted) and chroot to the hard drive root:

# umount /tmp/drivers
# chroot /mnt/sysimage

mount our updated cciss.o driver disk and overwrite the old SME Server kernel modules with ours (why I choose to create and populate such a directory structure should begin to be clear ;-)):

# mount -t auto /dev/fd0 /mnt/floppy
# tar xvzf /mnt/floppy/cciss.o.tgz

Check out the kernel modules dependencies are correct:

# /sbin/depmod -a 2.4.20-18.7
# /sbin/depmod -a 2.4.20-18.7smp

there should not be any output.

Quite finished, hold on... Let's create the updated ram disk boot images:

# cd /boot
# mv initrd-2.4.20-18.7.img initrd-2.4.20-18.7.img.original
# mv initrd-2.4.20-18.7smp.img initrd-2.4.20-18.7smp.img.original
# /sbin/mkinitrd initrd-2.4.20-18.7.img 2.4.20-18.7
# /sbin/mkinitrd initrd-2.4.20-18.7smp.img 2.4.20-18.7smp

then update the hard drive boot sector and say goodbye to the console:

# /sbin/lilo -v
# exit

Press CTRL-ALT-F1 to go back to the installer Complete notice, take a big breath, cross the fingers... and press OK at last.

The system will reboot: extract both the CDROM and the floppy disc from the drives, and let the SME Server Admin console start in its magnificency. Configure the system, reboot and...


The end?

Mostly.

I suggest editing the /etc/lilo.conf to speed up the boot: if you change the timeout value to 10 instead of the original 50 the LILO boot selector will flash for about 1-2 seconds, enough to press a key and change a selection, but sparing a great amount of boot time on a system that is inherently slow to start, due to all the initial messages and options prompt.

If yours is a multiprocessor system, change the default kernel LILO loads: edit the /etc/lilo.conf file and change the default value to SMEServer-smp.

If you changed anything in /etc/lilo.conf, update the hard drive boot sector:

# /sbin/lilo -v

If you want, you can copy this How-to to the /SME-6.0.1-01-cciss directory and then burn it on a CDROM for future references, just in case...

A quirk...

The tg3 network device driver works fine with the onboard Broadcom based gigabit network cards, but I noticed that upon reboot the NICs speed is 10Mbps, Half duplex even if autonegotiation is on and the devices they are connected to are faster (and autonegotiating in turn, too).

If you go to a root console prompt and restart the network services with the commands

# service network stop
# service network start

or, more simply, with

# service network restart

the NICs autonegotiate the right speed/duplexing.

Investigation is still pending about this phenomenon...

HTH - Pierluigi Miranda