Installing RedHat 9.0 on a Soekris 4801 SBC.

Introduction

A Soekris 4801 Singe Board Computer makes a perfect firewall / router. But as all my systems run some kind of RedHat based linux versions, I preferred to get RedHat running on the soekris box too, instead of a special linux or *BSD version. In the following I will explain the steps I took to get my system installed.

Outline

  1. Install a minimal Redhat 9.0 on a spare system.
  2. Configure that system that it can run with only a serial console
  3. Make it as small as possible, removing unneeded files.
  4. Copy that system over to an NFS server, to be exported as NFS root.
  5. Configure a PXEboot configuration for the Soekris system, using the new tree as NFS root.
  6. Boot the soekris over PXE, using NFS root.
  7. Partition the CF disk, and make a filesystem on it, then copy the FS from the NFS server to the CF disk.
  8. Boot over PXE, but now using the CF disk as root.
  9. Install the GRUB bootloader, and reboot from CF disk.
  10. Tune your config in a way that you can run with a read-only mounted CF disk.

Details

1. Install a spare system

Install RedHat on a spare system. It is much easier to install with a graphical console. Choose 'Custom Install', and then select a minimal configuration. Don't use a /boot partition, only a root partition. No swap, because we will not be swapping to the CF disk in the soekris. This should be a straightforward installation.

2. Configure the minimal system

Connect a laptop (or any system with a serial terminal emulator) to the spare system, using the serial line, and then configure the new system to use the serial line as console. Therefore theses files must be changed:
/etc/inittab:
...
# Don't run gettys on VT's
#1:2345:respawn:/sbin/mingetty tty1
#2:2345:respawn:/sbin/mingetty tty2
#3:2345:respawn:/sbin/mingetty tty3
#4:2345:respawn:/sbin/mingetty tty4
#5:2345:respawn:/sbin/mingetty tty5
#6:2345:respawn:/sbin/mingetty tty6

# Use the serial port as console
s1:2345:respawn:/sbin/agetty -L 19200 ttyS0 vt100

In /boot/grub/grub.conf add:
serial --unit=0 --speed=19200 --word=8 --parity=no --stop=1
terminal serial

Now try rebooting the system using only the serial port.

3. Remove unneeded files

A standard RedHat install will install all doc's and man-pages of all packages selected. These can be easely removed. For a minimal system, I also remove all native language support. Here is my list to remove:

rm -rf /usr/share/doc
rm -rf /usr/share/i18n
rm -rf /usr/share/info
rm -rf /usr/share/locale
rm -rf /usr/share/man
rm -rf /usr/lib/locale
rm -f /etc/sysconfig/i18n

You can also remove the vim-common package, although it will complain about dependancies.

4. Copy the FS over to a server

I will not go into the details of setting up a NFS root server, because that is not soekris specific, and there are lot's of pages containing this information. I assume that server:/export/soekris is exported, and accessible from the soekris and the spare system.
spare# mount server:/export/soekris /mnt
spare# cd /
spare# tar clSf - . | (cd /mnt; tar xpSf -)
spare# umount /mnt
The S options to tar are to ensure efficient handling of sparse files like libraries.

5. Configure PXE-Boot server

Using a standard PXE Boot server setup, here are the relevant parts of the config files:
/etc/dhcpd.conf:
        host soekris {
            hardware ethernet <mac of your soekris here>;
            filename "/tftpboot/pxelinux.0";
            option root-path "/export/soekris";
            fixed-address soekris;
    }

/tftpboot/pxelinux.cfg/<ip-in-hex>:
SERIAL 0 19200 0
PROMPT 1
DEFAULT cf
TIMEOUT 40
label cf
kernel soekris-2.4.26
append console=ttyS0,19200n81 root=/dev/nfs nfsroot=<server-ip>:/export/soekris \
 ip=dhcp panic=10 ramdisk_size=16384 rw

This assumes a usable kernel image in /tftpboot/soekris-2.4.26.  You can compile a customized kernel for the soekris, or use the redhat kernel. We still have all the redhat modules on the NFS-root image. The Ramdisk parameter is not yet needed, but will be in the next steps. In order to boot over NFS-root, we have to temporary modify the /etc/init.d/network script: it should not mess with the eth0 interface (don't try get a new DHCP lease on boot, don't shutdown eth0 on shutdown or reboot). Just comment the relevant lines out.
Also modify the /etc/fstab (on the server: /export/soekris/etc/fstab):
/dev/root               /                       nfs     defaults        1 1
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /proc                   proc    defaults        0 0
none                    /dev/shm                tmpfs   defaults        0 0

6. Boot the soekris diskless

Now we connect a terminal emulator to the soekris,  connect it to your network, and power it up. If you have a CF card installed, you must use <CTRL-P> to get to the bios prompt. Then try PXE booting with the 'boot F0' command.  If it doesn't work, try tcpdump on your server to find out what is missing. Also watch /var/log/messages on your server. If you get in trouble during multiuser startup, you can add the keyword 'single' to the pxelinux.cfg/<ip-in-hex> file. This should give you at least a single user prompt.

7. Install on the CF disk

Now we have a running system, running from NFS-root, but with a local /dev/hda on the CF card. We can proceed partitioning the CF disk, using fdisk, creating one single partition. Create a (ext2) filesystem on that partition with mk2efs. Then mount the CF disk on the NFS root, and copy the FS over to the CF disk:
soekris# fdisk /dev/hda
soekris# mke2fs /dev/hda1
soekris# tune2fs -c -1 /dev/hda1
soekris# mount /dev/hda1 /mnt
soekris# cd /
soekris# tar clSf - . | (cd /mnt; tar xpSf -)
Now change the fstab on the CF disk (/mnt/etc/fstab) to reflect the correct location of the root:
/dev/hda1               /                       ext2     noatime        1 1
We also change defaults into noatime, to reduce the amount of writing to the flash.
Now shutdown the soekris, and proceed to the next step.

8. Boot over the net, using the CF disk as root

Change in the /tftpboot/pxelinux.cfg/<ip-in-hex> the append line into:
append console=ttyS0,19200n81 root=/dev/hda1 ip=dhcp panic=10 ramdisk_size=16384

Now you should boot the soekris again, and it should boot over the net, but using the local CF disk as root.

9. Install the GRUB bootloader on the CF disk

If this boot is successfull, you can now install the bootloader on the CF disk, in order to be able to boot directly from the CF disk.
soekris# grub
grub> root (hd0,0)
grub> setup (hd0)
grub> quit

You should now undo the changes to /etc/init.d/network, because we now have to startup eth0 ourselves. Now we can reboot, and the soekris should boot from the local CF disk.

10. Tune your config to run with a read-only mounted root FS.

The first thing you need to run with a read-only root FS is a ramdisk to store all the things that normally would go to the root fs. Depending on your applications, a 16Mb ramdisk should be sufficient.  I have modified the startup scripts a lot, and made changes to the filesystem to run read-only. Here is a brief overview of the things involved:

  1. Create a RAMDisk in the rc.sysinit. I added it instead of the remount of the root fs to read-write.
    mke2fs -F -q -m 0 /dev/ram0 2>/dev/null
    mount -n -t ext2 /dev/ram0 /rd
  2. Create the directory tree on the ramdisk that is needed to hold the read-write files:
    mkdir -p /rd/var/run/netreport /rd/var/log /rd/var/lock/subsys /rd/tmp /rd/etc \
                  /rd/dev /rd/mqueue /rd/clientmqueue /rd/mail /rd/var/net-snmp /rd/etc/ntp \
                  /rd/anacron
    Correct the modes/ownerships for the tree:
    cp -rp /etc/ntp.static/* /rd/etc/ntp
    chmod 700 /rd/mqueue
    chmod 777 /rd/tmp
    chmod o+t /rd/tmp
    chmod 770 /rd/clientmqueue
    chown smmsp:smmsp /rd/clientmqueue
    chmod 775 /rd/mail
    chown root:mail /rd/mail
    chmod 700 /rd/var/net-snmp
    touch /rd/var/log/lastlog

    (put these commands in rc.sysinit too).
  3. Rename some directories to a 'static' version, and create a symlink from the original location to the ramdisk version:
    mv /etc/ntp /etc/ntp.static
    ln -s /rd/etc/ntp /etc
  4. Remove some directories in /var, and replace them with symlinks to the ramdisk:
    rm -rf /var/run
    ln -s /rd/var/run /var
    rm -rf /var/log
    ln -s /rd/var/log /var
    rm -rf /var/lock
    ln -s /rd/var/lock /var
    rm -rf /var/spool/mail
    ln -s /rd/mail /var/spool
    rm -rf /var/spool/mqueue
    ln -s /rd/mqueue /var/spool
    rm -rf /var/spool/anacron
    ln -s /rd/anacron /var/spool
    rm -rf /var/net-snmp
    ln -s /rd/var/net-snmp
  5. As we cannot use a /etc/mtab (on a read-only filesystem), we create a dummy mtab file with the correct contents, and place it in /etc/mtab.static. Then we add a cp command to rc.sysinit to copy it to the ramdisk, and add a symlink to /etc. In this way, we assure the correct contents of /etc/mtab, even if the system is shut-down with a writable root fs:
    rm /etc/mtab
    ln -s /rd/etc/mtab /etc
    In rc.sysinit:
    cp /etc/mtab.static /rd/etc/mtab
  6. Add mount commands for special filesystems to rc.sysinit: Those filesystems must be mounted before you issue the cp command for the mtab in the previous step (otherwise it will think those filesystems are already mounted).
    mount -n -t usbdevfs usbdevfs /proc/bus/usb
    mount -n /dev/pts
    mount -n /dev/shm
  7. Comment out the lines in the rc.sysinit where the root fs is remounted read-write.
  8. Reboot the system, and check the output of the shutdown and boot for any error messages. That indicates other open issues. Things I had to change:
This should give you the general idea. Depending on the functions you want your Soekris to perform, your solutions may vary from mine. If you have any comment, suggestions, or questions regarding this web page, please contact karel@vhouten.xs4all.nl.

(c) Karel van Houten 2004