Virtualization: Xen (Part 1)


A Little Xen:

This blog entry is going to be a bit long. You’ve been warned. 🙂 I’ve been using virtualization on x86 hardware since 1997/98 when I got my first copy of VMWare Workstation. I used PC emulators before on other platforms, but I could tell this was different. The performance was infinitely better than any emulators I’d ever used and I was suitably impressed. Since that time, I’ve also looked at other virtualization programs like Virtual PC on the Mac and later Windows, QEMU on Linux and Windows hosts, and within the past three years, I’ve solidly focused my attention on Xen. This is largely because of their completely unique approach to virtualization and the abilities Xen offers. Abilities that nothing other than VMWare’s high end product, ESX, offers.

The thing that led me to Xen was that renewing the VMWare Workstation license was becoming prohibitive and unjustifiable for hobbyist use. (That’s changed since then but I’m into Xen now.) However, I still needed something to virtualize Windows XP since I no longer had to run it on bare metal for my needs. I’m not a gamer, and audio and video production tools for Linux are currently considerably better than they used to be, but I still need occasional access to “Windows only” stuff. At this time (2004) I was using QEMU with it’s accelerator but I wanted something better. While looking around on the net for other free virtualization systems I found the Xen project. I decided to install it on my Fedora Core 2 box and see what it would do. It didn’t start off that well because I got the Xen hypervisor (microkernel) to boot, but the console would then stop giving me any output or way to interact. I assumed it wasn’t working until I pinged the IP of my Fedora Core 2 box. It responded! I could ssh in! Weird. It was running, but no output on the screen. What else can I say other than I was intrigued.

Class Time:

Before I move forward, some background on Xen and a little vocabulary lesson is in order so I can communicate the Xen difference a little better. Most virtualization software usually requires that you install a “Host” operating system and then once that’s up and running, you install the virtualization platform itself and finally you install one or more “Guests”. While this works, quite well I might add, there is generally a performance hit because of the intermediate host OS. At the core of the virtualization process, there is a mixture of approaches to get hardware access from the guest redirected to either real hardware on the host or virtual hardware in software. Again, this can cause some reduction in performance as well as impose some limitations on hardware access itself. The way most virtualization techniques try to work around this is by translating the guest OS calls from a higher level of execution (ring 1, where applications usually run) to the host OS’s lower level of execution (ring 0, where the OS usually runs). This is a gross oversimplification by the way, but it will work to generally explain what’s going on.

The main difference with Xen vs. other virtualization approaches back then was that Xen attempts to create a virtual replacement for the x86 CPU. Instead of booting a host OS and then running Xen, you instead boot the Xen microkernel which then loads the first virtual machine. The Xen microkernel is very small and layers on top of the CPU, RAM, storage, network and peripheral buses to manage access for all virtual machines. So let’s get to our Xen definitions:

Paravirtualization: A system for managing access to the hardware resources of a computer system. The microkernel sits between the virtual machines and the hardware. In order for this to work, each virtual machine’s OS kernel needs to be compiled for the Xen virtual CPU target. The main benefit of this approach is that you can get almost 100% bare metal speed for all the virtual machines because they are essentially sharing the bare metal. The Xen microkernel creates something akin to a “ring -1” and the virtual machines can then run in ring 0. (Obviously this won’t work for Windows, but I’ll get to that later) Another nice benefit is that you can isolate and reserve physical resources like PCI and USB devices and assign them to different virtual machines. For example, a PCI video capture card can be assigned to one virtual machine, while a second one could be assigned to another.

Domain: In Xen, there is no such thing as a guest OS since there is no OS that the virtual machines are running on top of at all. Instead the virtual machines are called domains. There is a required primary domain called Domain0 that you use for managing your other domains, but it’s not a guest.

PV Domain: These are domains that are paravirtualized. Currently you can only paravirtualize OSes that you can recompile the kernel on. That limits PV domains to Linux and the BSDs. But PV domains are the most powerful mode of operation, so worth exploring if you use one of these platforms.

HVM Domain: When Xen 3.x came out they introduced new support for hardware assisted virtualization using Intel and AMD’s comparable virtualization technologies. To do this, they incorporated a modified version of QEMU into Xen. This allows you to run OSes like Windows or older Linux distros that don’t have a 2.6 series Linux kernel in Xen. You will need a newer CPU that supports Intel Vanderpool or AMD Pacifica technologies though. There is a drawback to HVM regarding steadily decreaing disk and network I/O performance.


It took me a little extra work, but I eventually figured my way around Xen and dedicated a box to using it. I then created paravirtualized Linux images and saw that the performance of my virtual machines was nearly 98% of running on the bare metal. It was really amazing, especially on such old hardware. My only limitation on the test box was RAM. If I had more RAM I could run many VMs on this old box. I got around to that and ended up with an old P II era celeron 400 with 384 Megs of RAM running three VMs very very well:

Domain0 (the management domain): Doing DHCP and NTP for my network. You typically shouldn’t have this domain doing much other than managing your VMs.
Domain1 (the “External” home server): Offering up three of my lame web sites, an SMTP smarhost, VPN services, external DNS for my domains
Domain2 (the “Internal” home server): Offering up MySQL for my dbmail installation, dbmail itself, postfix SMTP for internal use, internal DNS

The box is great for being so old and the virtualization really adds an extra layer of security that is unsurpassed (not perfect though). In fact, the Xen project’s technology was so impressive that MS itself is using the technology for their upcoming hypervisor in Longhorn called HyperV.

Enter HVM Domains:

With the introduction of Xen 3.x I wanted to try HVM domains. This was not possible until the first CPUs from Intel and AMD were released with hardware virtualization support. In August 2006 I bought a cheap Athlon 64 HP box at Best Buy and packed it with 4 gigs of RAM. I installed Xen 3.0 and either migrated from the old Celeron or set up new VMs on this box. There really isn’t much of a limit other than RAM and storage as far as I can tell. Right now, this box is running a Gentoo build VM, an Asterisk PBX VM, two domains migrated from the old Celeron and one more VM migrated from QEMU. Until December 2007 I also had a Windows XP VM (which I’ve actually moved back to QEMU for different reasons) running on this box as well.

All of the domains perform well and there is a mix of PV and HVM domains. The only issue I’ve seen on busy HVM domains is a gradual steady decrease in performance with network and disk I/O. This is something that I’ve seen others note was well. The simplest explanation I’ve found is that the virtual NIC and IDE interfaces are constructs in Domain0’s RAM. The busier the I/O, the more likely it is that the RAM will top out and you’ll get a performance hit due to swapping or waits (until RAM is free). Thus my sparing use of HVM with the free version of Xen.

What Makes Xen Really Different Though?

I thought you’d never ask. What else is there that sets Xen apart from the others (VMWare ESX excepted)? One feature is called “live migration”. If you have two identically configured physical hosts running the Xen microkernel, centralized storage on a third box for your VM images, and on the same network, you can actually move the VM between the two physical boxes with no downtime at all. All services continue to run, users don’t get bumped off, not even a slight pause of any kind at all. If you make sure that you don’t go beyond a reasonable amount of resource usage, on both physical hosts, this would allow you to move all the VMs from one physical box to another for actual hardware downtime/maintenance and no loss of service. How cool is that? For Linux domains, you can also change the amount of RAM allocated to a domain (as long as you’re running PV domains) while the system is live. Expand or shrink the RAM and then move it to another VM as needed. I’ve done this on the old Celeron a few times and it really is quite amazing.

If you use PV domains, the box can run headless and you can access the VM consoles with the Xen ‘xm’ management tool via an ssh session. All of the Xen management tools are CLI based stuff, so ideal for the CLI fans out there. Newer version of Fedora and CentOS come with Xen built in as an option, but I had a little trouble using these since they try to layer slightly immature GUI tools on top of Xen as well as change the way Xen’s VM configuration files work. If you run HVM domains, then you have the option of using VNC or SDL for your VM’s console display. VNC allows for network connection to the console, SDL does not. This is inherited from QEMU, so if you are familiar with QEMU it’s identical.

Is Anyone Actually Using Xen in Real Life?

Yes. As Xen has matured it spawned two commercial entities who made use of the technology. The first and most notable is Xensource. This company was started by one of the original founding members of the Xen project. They were acquired by Citrix in 2007. They support paravirtualization and provide special drivers for Windows that allow you to paravirtualize the OS without needing to rebuild the kernel. At least that’s my understanding. The other product is VirtualIron which is what I’m more familiar with in my work environment. They decided to go with HVM domains instead and offer special drivers to optimize the disk and network I/O performance. As a result they only formally support Windows and a small set of Linux distributions. They recently released the source for their VSTools package (drivers and utilities) so one can build support for unsupported distros but this is not a trivial task to complete. I’m working on building VSTools for Gentoo right now and it’s leading me down the path of building a custom initramfs. Fun for me, but not for most. I’ll be posting my experience with VirtualIron in a production setting in my next entry later today.


No Responses Yet to “Virtualization: Xen (Part 1)”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: