CentOS 5 Xen

Creating initrd files for CentOS 4.x domUs

CentOS 4.x seems to have a slightly different boot process than CentOS 5.x, so initrd files generated under CentOS 5 don’t really work that well. Here’s my procedure for generating initrds that work with CentOS 4.x domUs.

In the CentOS 4.x domU, make the proper directory for the kernel modules to live in…

mkdir -p /lib/modules/2.6.18-8.1.6.el5xen/

From the host, rsync the kernel modules into the domU…

rsync -avHlx -essh /lib/modules/2.6.18-8.1.6.el5xen/ root@domU:/lib/modules/2.6.18-8.1.6.el5xen/

In the domU, make the initrd…

mkinitrd -f -v /tmp/initrd-2.6.18-8.1.6.el5xen.img 2.6.18-8.1.6.el5xen --with=xennet --with=xenblk --with=sd_mod

From the host, copy the initrd file so it can be moved into place…

scp root@domU:/tmp/initrd-2.6.18-8.1.6.el5xen.img /tmp
mv /tmp/initrd-2.6.18-8.1.6.el5xen.img /boot/initrd-2.6.18-8.1.6.el5xen.centos4domU.img

I also use a couple of symlinks to link the newest kernel/initrd to a generic name, so I don’t have to change my domU configuration files all the time…

cd /boot
ln -s initrd-2.6.18-8.1.6.el5xen.centos4domU.img initrd-current.el5xen.centos4domU.img
ln -s vmlinuz-2.6.18-8.1.6.el5xen vmlinuz-current.el5xen

Configuring xen bridges on multiple interfaces automatically

The easiest way I’ve found to automatically configure xen bridge interfaces is to modify the xen script that starts the network bridges.

mv /etc/xen/scripts/network-bridge /etc/xen/scripts/network-bridge.xen
touch /etc/xen/scripts/network-bridge
chmod +x /etc/xen/scripts/network-bridge

Now, place the following contents into /etc/xen/scripts/network-bridge

#!/bin/sh
# Exit if anything goes wrong.
set -e

# First arg is the operation.
OP=$1
shift

script=/etc/xen/scripts/network-bridge.xen 

case ${OP} in
  start)
        $script start vifnum=0 bridge=xenbr0 netdev=eth0
        $script start vifnum=1 bridge=xenbr1 netdev=eth1
        #$script start vifnum=2 bridge=xenbri netdev=dummy0
        ;;

    stop)
        $script stop vifnum=0 bridge=xenbr0 netdev=eth0
        $script stop vifnum=1 bridge=xenbr1 netdev=eth1
        #$script stop vifnum=2 bridge=xenbri netdev=dummy0
        ;;

    status)
        $script status vifnum=0 bridge=xenbr0 netdev=eth0
        $script status vifnum=1 bridge=xenbr1 netdev=eth1
        #$script status vifnum=2 bridge=xenbri netdev=dummy0
        ;;

    *)
       echo 'Unknown command: ' ${OP}
       echo 'Valid commands are: start, stop, status'
       exit 1
esac

In order to add more bridges, you’ll just need to add more lines that correspond to the appropriate physical interface in each section.

Configuring bridges for multiple VLANs

This section will be beefed up a bit later. I’m rushing it out due to popular demand!

The default Xen networking scripts that come with CentOS 5 aren’t friendly with VLAN bridges. It’s explained somewhat thoroughly here, so I won’t repeat myself. Here’s the nitty-gritty on how to get around it.

First, you’ll want to edit your network-bridge script. I found a more elegant solution that uses a bash function to overload (and mask) the ifdown bash script somewhere out on the interweb, but I can’t find it right now. This was the solution I came up with. Look for a function definition for the “is_bonding()” function call. On my installs, it’s on line 78. The one line contained in the function should look like this by default:

[ -f "/sys/class/net/$1/bonding/slaves" ]

You’ll want to change this to the following:

[ -f "/sys/class/net/$1/bonding/slaves" ] || [ -f /proc/net/vlan/$1 ]

This will bypass the broken ifdown call that destroys the VLAN interface when the bridge is being created.

Second, you’ll want to make/modify a network initialization script. I chose to make my own, and modify the xend config file (/etc/xen/xend-config.sxp) to reflect the location of my network init script. I called my script “network-multi”, so I needed to find the network-script configuration directive and replace it with this:

(network-script network-multi)

Now that xen is looking for my network init script instead of the default, it’s time to make it. Here’s the important stuff from the script.

#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge="xbr_trunk"
ifup vlan10
ifup vlan11
ifup vlan12
"$dir/network-bridge" "$@" vifnum=1 bridge="xbr_vl10" netdev="vlan10"
"$dir/network-bridge" "$@" vifnum=2 bridge="xbr_vl11" netdev="vlan11"
"$dir/network-bridge" "$@" vifnum=3 bridge="xbr_vl12" netdev="vlan12"

And that’s it. The above assumes that you have three VLAN interfaces defined that will start up “attached” to eth0. Since they’re starting after the “xbr_trunk” bridge is created, they’ll be attached to the virtual eth0 interface instead of the physical ethernet device. This script doesn’t yet behave very nicely on shutdown, but that doesn’t bother me too much.

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>