Convert a CentOS 5 HVM domU to PV with PyGrub

Problem

Let’s say you’ve got a fully-virtualized (HVM) CentOS 5 Xen instance, but you want to convert it to run in a paravirtualized (PV) state. There’s a good number of reasons to do so, mainly for increased performance and stability. You also want to be able to upgrade kernels within the instance without having to make changes on the parent. Thankfully, it’s pretty easy to do. This how-to assumes that you’re already using PV disk drivers in your domU, so if you’re not, you may have a few extra steps that aren’t covered.

Solution

This task mainly centers around getting the kernel-xen RPM installed, and properly configuring the OS environment to use it, and to continue using it. This first set of steps takes place within your domU instance.
First, make necessary backups of your domU instance and dom0 configs to your satisfaction. Why, you ask? You can never be too sure. Just do it. Next, you’ll want to make sure that the process that creates initrds will properly include the PV disk modules.

mkdir -p /etc/sysconfig/mkinitrd/
echo 'MODULES="xenblk"' > /etc/sysconfig/mkinitrd/xen
chmod +x /etc/sysconfig/mkinitrd/xen

Next, we install the PV-compatible kernel. Feel free to omit the kernel-xen-devel package if you don’t require it.

yum -y install kernel-xen kernel-xen-devel

At this point, you should check the domU’s grub config. It’s likely that the RPM scripting in the kernel-xen package thinks your domU is a “normal” hardware node, and installed the kernel-xen package as if it were a dom0 node. If the newly-created section for your Xen kernel contains “xen.gz”, like in the example below, your domU will not boot and will need to be fixed in the next step.

title CentOS (2.6.18-308.4.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-308.4.1.el5
        module /vmlinuz-2.6.18-308.4.1.el5xen ro root=LABEL=/ hda=none
        module /initrd-2.6.18-308.4.1.el5xen.img

To repair this, run the following command, substituting your kernel version for the one I include below.. If you don’t see “kernel /xen.gz-[version]” line, you should be able to skip this step.

/sbin/new-kernel-pkg --package kernel-xen --mkinitrd --depmod --install 2.6.18-308.4.1.el5xen

This will perform the magic needed to boot the Xen-compatible kernel, and replace the grub config with a good version. When this step finishes, the section for your newly-installed kernel should look something like this.

title CentOS (2.6.18-308.4.1.el5xen)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-308.4.1.el5xen ro root=LABEL=/ hda=none
        initrd /initrd-2.6.18-308.4.1.el5xen.img

If you value being able to use the xen console to manage your domU, you should add the following to /etc/inittab. Look for the line similar to the following:

6:2345:respawn:/sbin/mingetty tty6

… and then add the following line after it.

7:2345:respawn:/sbin/mingetty xvc0

The next step involves editing the config file on your dom0 node to use pygrub. The dom0 configuration I use has a source-compiled version of Xen instead of the stock RPM version of Xen. You may need to go looking for your copy of pygrub if it’s not int the same location. Edit your domU configuration file, and replace the following two lines …

kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'

… with these two lines …

kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra = "(hd0,0)/grub/menu.lst"

This covers a 64-bit domU. If your domU is 32-bit, use the following.

kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"
extra = "(hd0,0)/grub/menu.lst"

At this point, you’ll want to fully shutdown your domU so that it can be booted with the new settings. Simply issuing a ‘reboot’ in the domU isn’t good enough. After it has been shut down, you’ll want to start it up again, with the Xen console attached, since you’ll have to boot the new kernel manually via pygrub and interact with the menu.

xm create /xen/configs/domU.cfg -c

Pygrub will spit out a lot of text when the domU first starts, but pay attention for when it gets to to the part where you can pick your kernel. You’ll (obviously) want to choose your new kernel-xen kernel from the menu. If all goes well, you should see the familiar linux boot sequence, and eventually you’ll get your login prompt (if you modified /etc/inittab that is). Assuming that everything has booted up properly, the next steps should be performed in your domU, and are mainly for cleanup.
You’ll want to alter your grub configuration so that your new kernel boots by default. In my case, this was easy as editing the /boot/grub/grub.conf file, and changing “default=1” to “default=0”. Your mileage may vary.
This will tell the system to default to kernel-xen kernels when doing system updates, instead of the normal non-xen kernels:

perl -pi -e 's/^DEFAULTKERNEL=kernel$/DEFAULTKERNEL=kernel-xen/g' /etc/sysconfig/kernel

Next, you’ll probably want to remove the old non-xen-compatible kernels. On my example system here, I had the following kernels installed.

[root@n13 ~]# rpm -qa | grep kernel
kernel-xen-devel-2.6.18-308.4.1.el5
kernel-devel-2.6.18-238.el5
kernel-devel-2.6.18-238.12.1.el5
kernel-2.6.18-238.12.1.el5
kernel-xen-2.6.18-308.4.1.el5
kernel-2.6.18-238.el5
kernel-headers-2.6.18-238.12.1.el5

With that in mind, I removed the following RPMs:

rpm -e kernel-2.6.18-238.el5 kernel-2.6.18-238.12.1.el5 kernel-devel-2.6.18-238.el5 kernel-devel-2.6.18-238.12.1.el5

It’s a good idea to examine the contents of your grub.conf file to make sure it is still sane.

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/hda4
#          initrd /initrd-version.img
#boot=/dev/hda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-308.4.1.el5xen)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-308.4.1.el5xen ro root=LABEL=/ hda=none
        initrd /initrd-2.6.18-308.4.1.el5xen.img

If your new kernel is still there, and is still default, you should be good to go!

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>