Installing & Configuring Xen (2.0-testing) under Linux

This document is a short practical guide through the configuration and installation process of Xen linux 2.6 host and guest domains. There are many other documents and FAQ's focussing on different platforms or specifics which can be found on the Xen Documentation homepage.

Building the kernels
Installing the kernels and tools
Creating & configuring a user (guest) domain DomU
Running & controlling a user domain
Network setup and options
X and domU

1. Building the kernels Download and extract the testing source code of xen from the Xen download page. The current version of Xen-Testing uses the 2.6.12 kernel and will download the source tarball when issuing make unless the file linux-2.6.12.tar.bz2 is found in the top level xen directory. To prevent the download simply copy the file over manually. The download server can be modified in the buildconfigs/Rules.mk file (search for 'kernel.org').
By default, the toplevel Makefile contains the line

KERNELS ?= linux-2.6-xen0 linux-2.6-xenU

which means that two linux 2.6 kernels will be built, a dom0 and a domU in their respective folders. This is probably what you want when building for the first time. Later when you are happy with one of the kernels, say dom0, but you want to change the configuration of domU you can change this line to

KERNELS ?= linux-2.6-xenU

to accelerate the build process.
For now, it is assumed that you build both, a -xen0 and a -xenU kernel.
To start the build process type

$ make world

This will do the following:

clean up any existing build files
extract the kernel source (download first if not found)
apply all kernel patches that are found in the 'patches' subdirectory
apply the Xen patches
create kernels, tools, and documentation inside the 'dist/install' directory tree

CAUTION: Only use make world if you are building the kernels for the first time because the world target will delete existing .config files! If you customized the kernels and you are simply rebuilding do make kernels or make install-kernels instead (see below)!

The kernels have now been built with the default configurations. To customize a kernel you can type

$ make -C <kerneldir> ARCH=xen menuconfig

where <kerneldir> is the subdirectory of the kernel you want to modify, eg. linux-2.6.12-xen0.
There are some xen specific issues to look out for when configuring the kernels:
Make sure that all kernels with direct hardware access are configured as Privileged Guest (domain 0). However, user domains should not have the Block-device backend driver or the Network-device backend driver enabled, so disable those for the domU's. Also make sure that you only enable the options/drivers you really need for domU's otherwise you might end up with kernel panics.
As an example assume that we have a bt878-based video grabber that we want to assign to our domU to control. In the kernel configuration for this domain we select (at least):

Privileged Guest (domain 0)
PCI Support
I2C Support, I2C device interface
Video for Linux, BT848
(if using sound as well:) ALSA, Bt87x Audio Capture

It is a good idea to select the drivers to be built in (instead of built as modules) to prevent having to copy the modules from the build system to the guest system after each rebuild. (In our example we even disabled the module loader altogether which has been suggested by other Xen users as well.)
When you are done configuring the kernels type make kernels to rebuild them. 2. Installing the kernels and tools Run

# make install

to install everything from the dist/install subfolder into the system (ie. xen including the control tools (xend, xm), the kernels that have been built and the documentation). If you don't have the Twisted networking framework installed you might have to run

# make install-twisted

first which downloads the necessary package and installs it.
You probably only need to make these targets once (like make world) and when refining the kernel configurations or adding user domains use

# make install-kernels

instead. This will build all kernels specified in the KERNELS ?= ... line (mentioned above) and install them without reinstalling the xen tools and documentation.
The kernels will all be installed in /boot and are called something like vmlinuz-2.6.12-xen0 or vmlinuz-2.6.12-xenU.
Now you can edit the grub menu.lst file and add a section similar to:

# Xen 2.0 testing (kernel 2.6.12)
title  Xen 2.0 testing [kernel 2.6.12]
root   (hd0,2)
kernel /boot/xen-2.0-testing.gz dom0_mem=262144 physdev_dom0_hide=(03:04.0)(03:04.1)
module /boot/vmlinuz-2.6.12-xen0 root=/dev/sda3 ro

Modify the parameters to suit your environment.
This configuration will load the Xen hypervisor which then loads domain0 specified by the module line. It will assign 256MB of RAM and the devices given by 03:04.0 and 03:04.1 (ie. the video and audio grabber) will not be available to dom0 (and can be used by a user domain instead, see below). Make sure to format the device ID's as given. Apparently omitting digits (03:4.0) will not work!
Before rebooting to confirm that domain 0 runs as expected disable the incompatible TLS libraries (see section 2.4.3 of the main Xen documentation):

# mv /lib/tls /lib/tls.disabled

Now boot into dom0. The rest can be done from there. 3. Creating & configuring a user (guest) domain DomU The easiest way is to create a dedicated partition for the user domain. In that case you can simply install your preferred linux distribution using that partition as the root partition.
Make sure not to install any kernel, kernel modules or boot loaders. Or simply delete the /boot folder and the /lib/modules/<version> folder afterwards (from this partition!).
Alternatively you can create a filesystem inside a file. To do so:

$ dd if=/dev/zero of=/path/to/file bs=1k seek=4096k count=1

$ mkfs -t xfs /path/to/file

This will create a 4GB sparse file (ie. it will initially only use 1kB and grow as needed) and format it with the XFS filesystem.
The problematic part is to install linux in this filesystem. There are bootstrappers for several distributions but if you don't have one you will have to install to a 'real' partition first and then after mounting the virtual filesystem

# mount -o loop /path/to/file /mnt/tmp

copy everything over from the 'real' partition

$ cp -ax * /mnt/tmp

To use this file for the user domain umount it first, then

# losetup /dev/loop0 /path/to/file

(you might have to replace loop0 with a free loop device, eg. loop1). This will bind the loop device to the file.
Before going ahead make sure to adjust /etc/fstab, and the network configuration inside the new filesystem (refer to section 5 below). Also copy the /lib/modules/<version>-xenU folder from the build system to the filesystem to make the kernel modules available and adjust the modules.conf file (if you chose to build modules). As described in the manual you should also delete all but one console entry from the /etc/inittab file. You may also want to disable startup scripts that won't work in the virtual environment (such as setting the hardware clock). Finally, check that the /lib/tls folder does not exist (rename to tls.disabled as in dom0).
Now you can create a configuration file for the user domain:
You can use one of the example configurations in the /etc/xen folder as a basis (ie. cp xmexample1 myFirstDomU).
Here is a sample config (there are more options, refer to the documentation for those):

kernel = "/boot/vmlinuz-2.6-xenU"
memory = 512
name = "myFirstDomU"
disk = [ 'phy:sda5,sda1,w' ]
pci = [ '03,04,0', '03,04,1' ]
root = "/dev/sda1 ro"
extra = "3"

The kernel and memory lines are self-explanatory, the name will be shown when issuing a xm list command.
In this example the partition /dev/sda5 will be made available to the domain under the name /dev/sda1 with read-write access. Note that this partition must not be mounted writable or used anywhere else while this user domain is running. This is also true for swap partitions (ie. you cannot use the same swap partition for dom0 and domU!). The line starting with pci = ... tells Xen that the given devices will be accessible by this domain. Note that bus, device and function are separated by a comma here. Also note that this line usually implies that the following has been done:

These devices have been hidden from Dom0 (by the physdev_dom0_hide line)
This domain was configured as a privileged guest (see section 1)

The root and extra fields will be appended to the kernel line (ie. use /dev/sda1 as the root partition and go to runlevel 3 upon start up).
If you want to use a file instead of a 'real' partition modify the disk line as follows:

disk = [ 'phy:loop0,sda1,w' ]

(You simply replaced sda5 by loop0) 4. Running & controlling a user domain Now as root you can issue

# /usr/sbin/xend start

This will start the xen daemon. You might want to put this in an rc.x file to start it up automatically.
And to start the freshly created user domain:

$ /usr/sbin/xm create myFirstDomU -c

where myFirstDomU is the file name of the configuration to use and the -c parameter tells xen to connect to the console of the created domain which means you can follow the boot process of the guest on your screen and log in afterwards. To disconnect press Ctrl+], to reconnect check the Id of the domain

$ /usr/sbin/xm list

then switch to the console of the guest like this:

$ /usr/sbin/xm console <Id>

or use the name specified in the config file:

$ /usr/sbin/xm console <name>

If you don't get any output at all it is likely that you configured the domain with physical device access but did not select privileged guest (domain 0). So go back and reconfigure the kernel.
If the user domain panics or doesn't react or you simply want to kill it issue

# /usr/sbin/xm destroy <Id>

or to shutdown cleanly (if possible):

# /usr/sbin/xm shutdown <Id>

The letter under the state field shown by xm list means:

r - running,
b - blocked,
p - paused,
s - shutdown,
c - crashed

5. Network setup and options By default a guest system will have one virtual network interface. This can be changed with the nics option in the configuration file. Xen provides two general ways of networking for the guest systems. In the default setup a network bridge is created on the host under the name xen-br0. It is automatically linked to eth0 and to the running guest systems.
Try

# brctl show

in dom0 to see the bindings (vif1.0 is the first virtual interface of the user domain with Id 1, vif2.0 belongs to Id 2 etc). If a DHCP server is available and the guest is set up to use it then the guest should automatically have access to the network. You need the linux bridge utils to use the network bridge.

NOTE: If you used sudo to start the xen daemon without having brctl in your PATH (ie. /sbin or /usr/sbin) xen will silently fail to create the bridge and your guest domain won't be able to connect to the network! Restart xend as root to resolve this problem.

The other possibility is to let dom0 act as a router for the guests. Configuration takes place in the files under /etc/xen/scripts. These are python scripts that are called when a guest is started or when vif's are initialized.

As an alternative approach the network can be set up using NAT so that the guests communicate with the internet over the IP address of dom0. To do that create a new network script under /etc/xen/scripts (call it network-nat):

set -e
OP=$1
shift

for arg ; do export "${arg}" ; done

bridge=${bridge:-xennet0}
netdev=${netdev:-eth0}
src=`ifconfig ${netdev} | grep inet | cut -d : -f 2 | cut -d \  -f 1`

show_status () {
    local dev=$1
    local bridge=$2

    echo '============================================================'
    ifconfig ${dev}
    ifconfig ${bridge}
    echo ' '
    ip route list
    echo ' '
    route -n
    echo '============================================================'
}

op_start () {
    if [ "${bridge}" == "null" ] ; then
        return
    fi

    echo 1 >/proc/sys/net/ipv4/ip_forward
    if ! brctl show | grep -q ${bridge} ; then
        brctl addbr ${bridge}
    fi
    ip addr add 192.168.0.1/24 dev ${bridge}
    ip link set ${bridge} up
    iptables -t nat -F POSTROUTING
    iptables -t nat -F PREROUTING
    iptables -t nat -A POSTROUTING -o ${netdev} -s 192.168.0.0/24 -j SNAT --to-source ${src}
}

op_stop () {
    if [ "${bridge}" == "null" ] ; then
        return
    fi
}

case ${OP} in
    start)
        op_start
        ;;

    stop)
        op_stop
        ;;

    status)
        show_status ${netdev} ${bridge}
        ;;

    *)
        echo 'Unknown command: ' ${OP}
        echo 'Valid commands are: start, stop, status'
        exit 1
esac

Now edit the xend configuration file /etc/xen/xend-config.sxp so that it contains these lines:

(network-script  network-nat)
(vif-bridge      xennet0)
(vif-script      vif-bridge)

This method was discussed in the xen-devel mailinglist. For more information refer to the Xen User Manual (section 7.2) and the network section of the XenFAQ's. 6. X and domU At the current stage of the xen development there is no possibility to run a 'real' X server under a guest domain. Instead, you could connect to the "remote" Dom0's X server to run X based programs. Or, if you just need to run an X program without seeing its interface you can use the Xvfb program which is part of current X11 installations. It provides an X server without the need for a framebuffer device and thus runs fine in a user domain. To replace the default X session manager on a Slackware domU by Xvfb on runlevel 4 you can replace the /etc/rc.d/rc.4 file with the following:

# Tell the viewers what's going to happen...
echo "Starting up virtual X server..."

if [ -x /usr/X11R6/bin/Xvfb ]; then
    exec /usr/X11R6/bin/Xvfb :1 2> /dev/null
fi

# error
echo
echo "You don't have Xvfb. Can't use runlevel 4 without it installed."
sleep 30

# All done.

Also make sure that the (single) console line in the /etc/inittab file includes runlevel 4, ie. it should read something like

c1:12345:respawn:/sbin/agetty 38400 tty1 linux

When launching runlevel 4 a virtual X server will be created on display :1 and the usual console login prompt will be shown. After logging into the console you can use any X programs on this display (without seeing the actual GUI). Note however that you need to set the DISPLAY variable manually! Either set and export it before starting an X program or put a line similar to

export DISPLAY=:1

in the bash profile (/etc/profile or ~/.bash_profile).

It is possible to use memory mapped files for the output (option -fbdir) which can be viewed with the xwud(1) program from a machine with a 'real' X server (eg. dom0). For more information on the quite useful options of Xvfb see the manual page Xvfb(1).

Last modified: 28/10/2005 by Cihan Altinay (cihan (at) uq.edu.au)

Contents