Fast Iteration of Xen Developping
For last two weeks I’ve been doing some experiments on Xen and HVM. I found that the
TOCTTOR, short for
Time of Coding To Time of Result, is too long, which is really annoying and disrupting. I tried to shrink the time and now I have successfully shrink the time to less than 30s on my 2010 MBP (with SSD). Here is how.
1. Using VMware for Testing
Yes, it is fast enough. VMware does a great job on performance improvement these years. Using it can significantly reduce the time of rebooting, from more than 60s to about 10s.
A good news is that VMware now
supports EPT emulation, so I can run Xen on VMware, and run HVM on Xen. It also
supports virtual serial port, which is essential for Xen debugging.
2. Using rsync for Compiling
I edit and compile the source code on another machine with powerful CPUs, and
rsync the binary to VMware for testing. The compiling time is not long since I only modify a small part of Xen. The compiling and rsyncing take about another 10s.
Another benifit of using rsync is that you can keep reading and modifying the code when rebooting the test environment in VMware. It’s another kind of parallelization.
3. Using ‘xl save/restore’ for HVM Creating
Since my test is issued through HVM, I have to create two HVMs every time. The command
xl save/restore can save a lot of time. Now the HVM creating time is shrinked from 30s to 5s.
4. Some Configurations
The developping environment is Debian 7. VMware version is workstation-9 on PC and Fusion-5 on Mac.
4.1 Compiling my own kernel
# apt-get install build-dep linux # apt-get install linux-kernel-3.2 # the code and patch is now in /usr/src # make localmodconfig # make menuconfig # add TAP/TUN and Xen device drivers # make -j8 # make modules_install # make install # mkinitramfs 3.2.46-rt67 -o /boot/initrd.img-3.2.46-rt67
A common problem is that the X window fails to run, with error like:
- (EE) XKB: Could not invoke xkbcomp(EE) XKB: Couldn’t compile keymapKeyboard initialization failed. This could be a missing or incorrect setup of xkeyboard-config.
A solution is to add
GRUB_CMDLINE_LINUX="nopat" in /etc/default/grub, as shown later.
4.2 Compiling my own Xen
# apt-get build-dep xen # apt-get install bridge-utils # make xen -j8 # make tools -j8 # make install-xen # make install-tools PYTHON_PREFIX_ARG= # update-grub # vi /etc/fstab # echo "xenfs /proc/xen xenfs defaults 0 0" # vi ~/.bashrc # add "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64"
4.3 Serial output for Xen on VMware
# /etc/default/grub of domain-0 GRUB_DEFAULT=8 GRUB_TIMEOUT=2 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all com1=115200,8n1 console=com1" GRUB_CMDLINE_LINUX="nopat"
Second, add a serial device in VMware setting. It’s easy.
4.4 Config file of HVM
kernel = "hvmloader" builder = 'hvm' memory = 256 name = "vm1" cpus = "1" vif = [ 'bridge=xenbr0' ] disk = [ 'file:/root/VMs/vm1.ubuntu-8.04.img,hda,w' ] boot = "c" sdl = 0 vnc = 1 vncpasswd = '' stdvga = 0 serial = 'pty' tsc_mode = 0
4.5 Serial console for HVM
console=tty1 console=ttyS0 and
terminal part in grub.
# /boot/grub/menu.lst in HVM default 0 serial --unit=0 --speed=115200 terminal --timeout=5 serial console timeout 1 title Ubuntu 8.04.4 LTS, kernel 2.6.24-26-generic root (hd0,0) kernel /boot/vmlinuz-2.6.24-26-generic root=UUID=e60586d4-53b2-4392-939 0-2c4a131c073d ro console=tty1 console=ttyS0 initrd /boot/initrd.img-2.6.24-26-generic
serial = 'pty' part in HVM config file is also essential for serial console.