Configuring Fedora to use a AoE root Filesystem

Introduction

This document is meant to be a quick how-to on how to take an existing Fedora installation present on local disks and migrate it to an ATA over Ethernet (AoE) target so that it can be booted over the network in a diskless fashion. Why would you want to do this you ask? My reasoning is that I have a server with a SCSI RAID5 array, which is much more reliable than a single IDE disk. Running a workstation off that array over the network helps ensure that I won’t need to deal with reinstallation due to disk failure. I also run my MythTV frontend in a diskless configuration, which is nice because I can place it right next to large speakers without having to worry about a magnetically-induced disk failure.

This document assumes that you have a working Fedora installation on a client computer, a seperate server machine that will export the AoE target, a method of running gPXE on the client computer (be it through a boot floppy/CD-ROM, PXE chain loading, etc), and a DHCP server that will provide gPXE with the necessary boot parameters. It should also work for other Red Hat operating systems as long as they have access to the aoe kernel module, which is present in 2.6.11 kernels or later. It can be installed seperately for older kernels.

Prepare the server(s)

Export the AoE target

I use LVM to provide the framework for my AoE targets since it’s flexible and allows for expanding/shrinking of the partitions it contains. Creating the LVM partition is outside the scope of this document. You can also use file-based disk images, but I can’t speak to their performance or reliability.

On the machine where you’re exporting the AoE target, run vbladed to provide the spot to get things set up. It’s a good idea to limit the allowed MAC addresses so that the client machine is the only machine that can access the target.

vbladed 11 0 eth0 /dev/LVM-RAID/aoe-hyperion-fedora -m 00:40:f4:76:1e:28

Set up DHCP

I use gPXE to utilize an AoE target as a root filesystem. On the machine I’m using in these examples, it is loaded from a bootable CD. Installing gPXE is ouside the scope of this document, but the documentation on their website is getting better every day. I’d check there for assistance.

To tell gPXE what target to use, modify your DHCP configuration so that it it is similar to the following.

host hyperion
{
        hardware ethernet 00:40:f4:76:1e:28;
        fixed-address 192.168.0.11; 

        filename "";
        option root-path "aoe:e11.0";
}

The “filename” and “option root-path” lines are what make things go.

Prepare the client

Initialize AoE

In order to access AoE targets from the machine you want to duplicate, you’ll have to load the aoe module.

modprobe aoe

If you set up the AoE target before loading the module, you shouldn’t need to do anything else. You should be able to see the target show up as a device visible in “fdisk -l” or dmesg. If it doesn’t show up, you may need to run aoe-discover (from the aoetools package) to discover the device.

Prepare the AoE target

In order to use the AoE target as if it were a normal disk, you’ll need to partition it. How to do this is beyond the scope of this document, but I’ll provide mine as an example.

[root@hyperion ~]
: fdisk -l

Disk /dev/etherd/e11.0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

             Device Boot      Start         End      Blocks   Id  System
/dev/etherd/e11.0p1   *           1          13      104391   83  Linux
/dev/etherd/e11.0p2              14          78      522112+  82  Linux swap / Solaris
/dev/etherd/e11.0p3              79        1112     8305605   83  Linux

Once the target is partitioned, you’ll want to format the partitions that will become / and /boot, and initialize the swap.

mke2fs -j /dev/etherd/e11.0p1
mke2fs -j /dev/etherd/e11.0p3
mkswap /dev/etherd/e11.0p2

Copy Files into AoE target

On your client machine, you’ll need to copy everything onto the AoE target. This example assumes you have both a / and /boot partition. Modify as necessary.

mount /dev/etherd/e11.0p3 /mnt
rsync -avHlx / /mnt
mount /dev/etherd/e11.0p1 /mnt/boot
rsync -avHlx /boot /mnt/boot

chroot into the AoE target

It’s a good idea to chroot into the partions in the AoE target to make sure you’re updating the right files, and so the originals are unmodified in case you need to boot back into the local OS.

chroot /mnt /bin/bash

Edit fstab

You’ll need to edit your /etc/fstab to reflect the proper devices. I didn’t have much luck with using labels, so I’ve got things hard-coded.

/dev/etherd/e11.0p3     /                       ext3    defaults        1 1
/dev/etherd/e11.0p1     /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/etherd/e11.0p2     swap                    swap    defaults        0 0

Patch mkinitrd

Currently, the mkinitrd script needs to be patched to properly support AoE. I made the following patch to correct this. I will probably submit it to RedHat to see if I can get it included in future versions of mkinitrd, but for now it requres a patch.

--- /sbin/mkinitrd.bak  2007-11-17 19:57:02.000000000 -0500
+++ /sbin/mkinitrd      2007-11-17 20:03:34.000000000 -0500
@@ -1438,10 +1438,21 @@
         handlenetdev $netdev
         emit $network
     done
 fi

+
+case $rootdev in
+/dev/etherd/e*)
+       emit "mkdir -p /dev/etherd"
+       emit "mknod /dev/etherd/discover c 152 3"
+       emit "echo Telling AoE to discover."
+       emit "echo omgdiscover > /dev/etherd/discover"
+       emit "sleep 5"
+       ;;
+esac
+
 emit_iscsi

 if [ "$scsi_wait_scan" == "yes" ]; then
     emit "insmod /lib/scsi_wait_scan.ko"
     emit "rmmod scsi_wait_scan"

To get things patched, copy the patch info above into a file. I use patch.diff in my example. Then, make a backup of your mkinitrd script and then patch it.

cp /sbin/mkinitrd /sbin/mkinitrd.bak
cat patch.diff | patch -p0

Configure mkinitrd environment

Placing the following two lines in /etc/sysconfig/mkinitrd will force the mkinitrd script to load the aoe and network modules and bring up the proper ethernet devices. Modify as necessary.

NET_LIST="eth0"
MODULES="aoe 8139too"

Build the initrd

Now that mkinitrd is ready to rock and roll, its time to make the initrd.

mkinitrd -f -v /boot/initrd-`uname -r`.img `uname -r`

Watch the output to make sure that your ethernet and aoe drivers are included.

Install grub

Getting grub to install properly onto the AoE target took a bit of voodoo when I made my attempt. It wouldn’t install directly to the etherd device, so I had to trick it into thinking it was installing onto /dev/sda. To do this, I edited the device.map file in /boot/grub to contain the following.

(hd0)     /dev/sda

Then I created hard links between the etherd device nodes and nodes for /dev/sda.

ln /dev/etherd/e11.0 /dev/sda
ln /dev/etherd/e11.0p1 /dev/sda1
ln /dev/etherd/e11.0p2 /dev/sda2
ln /dev/etherd/e11.0p3 /dev/sda3

At that point, I was able to install grub from the grub command line in a normal fashion.

root (hd0,0)
setup (hd0)
quit

Edit grub.conf

Now we need to tell grub to look at the proper device for the root filesystem. Adjust the ‘root=” parameter to the appropriate value.

title Fedora (2.6.23.1-10.fc7)
        root (hd0,0)
        kernel /vmlinuz-2.6.23.1-10.fc7 ro root=/dev/etherd/e11.0p3 vga=771
        initrd /initrd-2.6.23.1-10.fc7.img

Modify startup services

I’m not sure why, but my test machine hung up when running kudzu upon boot. So, to avoid this issue, I’d recoommend turning it off until you’ve stablized things.

chkconfig kudzu off

Also, the network service needs to be on all the time, lest the computer lose its connectivity to the AoE export.

chkconfig --level 0123456 network on

Reboot

Now that all the necessary modifications are complete, you can reboot!

Modify BIOS settings

You may need to modify your BIOS settings to make sure that gPXE runs before the BIOS attempts to boot from your local disk. Modifying the BIOS boot order is outside the scope of this document, but if you’re hitting grub before you see gPXE output, you’ll need to make changes.

Great Success!

If everything went well, you should be sitting in your network-booted OS and can unplug the local disk at your leisure. If it didn’t boot properly from the network, your original OS should still be present on your local disk, so you can boot back into it and troubleshoot things.

Reference

  1. Nice tutorial ! thanks

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>