Fast Iteration of Xen Developping

For last two weeks I’ve been doing some experiments on Xen and HVM. I found that the TOCTTOR, short for Time of Coding To Time of Result, is too long, which is really annoying and disrupting. I tried to shrink the time and now I have successfully shrink the time to less than 30s on my 2010 MBP (with SSD). Here is how.

1. Using VMware for Testing

Yes, it is fast enough. VMware does a great job on performance improvement these years. Using it can significantly reduce the time of rebooting, from more than 60s to about 10s.

A good news is that VMware now supports EPT emulation, so I can run Xen on VMware, and run HVM on Xen. It also supports virtual serial port, which is essential for Xen debugging.

2. Using rsync for Compiling

I edit and compile the source code on another machine with powerful CPUs, and rsync the binary to VMware for testing. The compiling time is not long since I only modify a small part of Xen. The compiling and rsyncing take about another 10s.

Another benifit of using rsync is that you can keep reading and modifying the code when rebooting the test environment in VMware. It’s another kind of parallelization.

3. Using ‘xl save/restore’ for HVM Creating

Since my test is issued through HVM, I have to create two HVMs every time. The command xl save/restore can save a lot of time. Now the HVM creating time is shrinked from 30s to 5s.

4. Some Configurations

The developping environment is Debian 7. VMware version is workstation-9 on PC and Fusion-5 on Mac.

4.1 Compiling my own kernel
# apt-get install build-dep linux
# apt-get install linux-kernel-3.2 # the code and patch is now in /usr/src
# make localmodconfig 
# make menuconfig # add TAP/TUN and Xen device drivers
# make -j8
# make modules_install
# make install
# mkinitramfs 3.2.46-rt67 -o /boot/initrd.img-3.2.46-rt67 

A common problem is that the X window fails to run, with error like:

  • (EE) XKB: Could not invoke xkbcomp(EE) XKB: Couldn’t compile keymapKeyboard initialization failed. This could be a missing or incorrect setup of xkeyboard-config.

A solution is to add GRUB_CMDLINE_LINUX="nopat" in /etc/default/grub, as shown later.

4.2 Compiling my own Xen
# apt-get build-dep xen
# apt-get install bridge-utils

# make xen -j8
# make tools -j8
# make install-xen
# make install-tools PYTHON_PREFIX_ARG=

# update-grub
# vi /etc/fstab # echo "xenfs /proc/xen xenfs defaults 0 0"
# vi ~/.bashrc # add "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64"
4.3 Serial output for Xen on VMware

Frist, modify GRUB_CMDLINE_XEN part.

# /etc/default/grub of domain-0

GRUB_DEFAULT=8
GRUB_TIMEOUT=2
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""

GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all com1=115200,8n1 console=com1"
GRUB_CMDLINE_LINUX="nopat"

Second, add a serial device in VMware setting. It’s easy.

4.4 Config file of HVM
kernel = "hvmloader"
builder = 'hvm'
memory = 256
name = "vm1"
cpus = "1"
vif = [ 'bridge=xenbr0' ]
disk = [ 'file:/root/VMs/vm1.ubuntu-8.04.img,hda,w' ]
boot = "c"
sdl = 0
vnc = 1
vncpasswd = ''
stdvga = 0
serial = 'pty'
tsc_mode = 0
4.5 Serial console for HVM

Add console=tty1 console=ttyS0 and serial terminal part in grub.

# /boot/grub/menu.lst in HVM

default         0

serial --unit=0 --speed=115200
terminal --timeout=5 serial console

timeout         1

title           Ubuntu 8.04.4 LTS, kernel 2.6.24-26-generic
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.24-26-generic root=UUID=e60586d4-53b2-4392-939
0-2c4a131c073d ro console=tty1 console=ttyS0
initrd          /boot/initrd.img-2.6.24-26-generic

Note, the serial = 'pty' part in HVM config file is also essential for serial console.