I’m in the middle of reading Jef Raskin’s (started a project in the 70s called Apple Macintosh) book, The Humane Interface. It’s a terrific read that tells us all about what’s wrong with software as it stands today. Although Jef died in 2005, his son Aza is carrying on the work because it’s just too good to leave unfinished. Aza formed a company called Humanized.com and their product is called Enso (Windows only for now). It sits between the user and the system and applies the speed, power and efficiency of the command line with the elegance of the GUI. The closest thing I’ve seen to it is Firefox Ubuiquity (Also written by Aza Raskin), only this applies to the entire system.  While it’s not perfectly in line with Jef Raskin’s ideals, it’s a pretty nice step to getting there.  Check out the link to the demo video above to get a peek into what could be possible.   Enso is a command line for the non-technical person.  Computers don’t have to be unfriendly.


Difficulty Level: Moderate

What is a Network Block Device and Why Would I Want One?

Let me start this entry out by explaining just what a block device is, in case you’re a newer Linux user and you’re not sure. I didn’t know what one was at one point and a quick explanation would have been helpful. In short, block devices are things like hard drives, flash drives, floppy disks, CD-ROMs and even DVDs. On a more complex level, they are devices which get their input/output in the form of data blocks of a certain size in bits or bytes. For the sake of this discussion, we’ll just be thinking of the devices I listed above.

The Linux kernel among it’s many modules (which can be thought of as drivers) has a particular module called ‘nbd’ which stands for Network Block Device. What this means is that you can take almost any block device and present it to the network. This differs from standard Windows file sharing, or Unix NFS in that you’re not presenting a set of files to the network, but the raw device itself. (iSCSI is another way of accomplishing the same thing with a greater degree of reliability and is accessible to Linux through the Linux iSCSI Target and Linux iSCSI Initiator projects.  NOTE: Think of the target as the server, and the initiator as the client.) Although there are other ways of doing this, network block device support is still nothing to ignore. It may not be robust, but it’s still highly useful for non-mission critical tasks where the expense of a SAN is unwarranted.

I originally found out about nbd when I was first starting to work with Xen virtualization. Their project documentation suggested that a convenient way of being able to store virtual machine images on a server was to use nbd support. When I understood that this was the way to a “poor man’s SAN” since Linux software RAID and LVM volumes could be exported with nbd, I then wondered, what else could I export? I decided to experiment and find out. While it’s not perfect because of some I/O control limitations, it’s still quite handy and simple to implement.

Making NBD Work for You:

The first step towards preparing your system for NBD is determining if you need to recompile your kernel.  Fortunately, most of the popular Linux distributions tend to build most of the optional support as kernel modules by default.  You can think of modules as “drivers” for different functionality and hardware support.  In most cases, you should have access to NBD support.  The simplest way to find out is to get to a shell prompt and type: ‘modprobe nbd’.  If you get no error and simply return to the prompt, then type: ‘lsmod | grep -v grep | grep nbd’.  If you see a line indicating that the nbd module is loaded, your kernel has NBD support built in.

However, if you get an error with the ‘modprobe’ command about the module not existing, or you don’t get anything in return for the ‘lsmod’ command, then it’s likely you’ll need to recompile your kernel.  Recompiling the kernel is beyond the scope of this blog entry, but I will link to some resources to get you started in the right direction at the end of this entry.

Three Components to NBD:

There are three components that make up the entirety of NBD for Linux.  The first is the kernel module which allows the kernel to provide an interface to the device for export to/from the network.  The second component is the ‘nbd-server’ application which handles the exporting of the device over TCP/IP.  And finally, the third part is the ‘nbd-client’ application which imports the device on another machine and presents it as /dev/nbX where ‘X’ is a number.  Depending on the distribution you use, you may be able to find a specific package to install the applications.  If not, there is always the source code from the main NBD project site.

Once you have the kernel module loaded and the applications built here is all you need to do to test it and see if it will work for what you need.  This is an example of exporting a raw unpartitioned IDE hard drive over TCP/IP and then importing it on a remote system:

1. On the system containing the hard drive, run the nbd-server command as follows  (Syntax: nbd-server -r <tcp port> /dev/xxx):

nbd-server -r 2000 /dev/hda

2. On the remote system where you wish to import the devicem run the nbd-client command as follows (Syntax: /nbd-client <ip address of system running nbd-server> <matching tcp port number> /dev/nbd0)

nbd-cliet 192.168.1.1 2000 /dev/nbd0

You should then be able to treat /dev/nbd0 on the system where you imported the device as if it were local.  Use ‘fdisk’ to partition the device, format it for use as a file system, or even a paging file.  I’ve used this successfully with remote flash drives, raw hard drives, partitions on hard drives, LVM logical volumes, and even DVD drives for playing movies on devices that don’t have DVD drives.

A few extra notes:

1. Depending on the kernel version, the NBD device nodes might be /dev/nbdX or it might be /dev/nbX where ‘X’ is always a number.

2. There is a one-to-one relationship between the exported device and your chosen TCP port.  That is to say, that if you use port 2000 for /dev/hda and want to export /dev/hdb simultaneously, you’ll need to increment to the next free port.

3. Before randomly chosing ports, it’s a good idea to take a look at the commonly used ports listed in /etc/services as well as run a ‘netstat -an’ to see what ports your particular systems are using.

4. The performance of the DVD export over an 802.11bg link is quite good after an initial buffer period for something like Xine.

5. It’s quite possible to use an imported NBD device as part of a mirror set if you want a pseudo instant copy on a separate machine, but it’s not highly recommended.

Recompiling the Linux kernel (which seems to be a dying art):

http://www.digitalhermit.com/linux/Kernel-Build-HOWTO.htm


What Is This All About?

Every pioneering, epic journey worth taking from point A to point B requires asking for directions, wrong turns, heading in the completely wrong direction, or even getting lost. After all, you’re the scout. The first person to be sent out with enough knowledge to try and find your way to the highly desired point B. Your chances of success are greatly improved if you have some important skills and tools to take along with you. With that introduction, my intended goal of sharing some of my knowledge and tools with you regarding the world of Linux distributions can begin. Of course, before I proceed I should make it clear that I’m no expert and won’t pretend to be one. I’m just a keen experimenter, and a fan of what Linux distributions have enabled me to do. With that I begin my “From Here to There” series.

I was originally going to write a blog post on using Network Block Devices in Linux, when I kept running into the same problem over and over. My hope was that I could write an article simple enough for anyone to grasp. However, each time I went back to the entry, I discovered that there was yet another thing that I take for granted, that many Linux users might not understand. Each time I attempted to remedy that situation with some general guidelines, I found that I had to cover exceptions for the multitude of distributions out there which would really make the blog entry far too long and far too complicated for my intended audience: Linux newbies, and the highly curious users of other OSes.

So I sat down for a while and tried putting my finger on how it is that I can move between the Linux distros fairly easily and get work done. I’m no computer expert. I’m not a programmer. I’m just a basic system admin who went from Mac (pre OS X) to Windows (3.1 through XP) and got involved with Linux in 1997 out of necessity. While thinking, I realized that there are many things that we computer folks take for granted about what we do normally that fall into the category of “missing knowledge” for the average user. Without that knowledge, understanding how to use a computer is next to impossible. Many people before me have written about people who simply learn how to use specific brands of software, and do not acquire the application literacy needed to be able to use any software of the same category (ie. word processors). That’s not what the “Here to There” series will be about.

Instead, it will be my intention to try and share some of the more basic skills that I take for granted, when I use computers, that others might be unaware of. Everyone who is at an intermediate level or higher, with regard to computer use, has skills like these. But we rarely if ever discuss them because to us they are second nature. So the “Here to There” series will attempt to get people from point A to point B with a clear understanding as to how and why the journey is worth it. Since my experience is largely with Linux and Free/Open software, that is where some of my focus will be. But, in all honesty and with only a few exceptions, there really is little difference between OSes and applications today. As a result, much of what I share should apply to many OSes and applications. I will be attempting to post the first in the series this week.  (Correction: Life, as usual, has become fairly busy.  So I will be starting the series sometime soon.  The first part should appear when it’s ready…)


Ingredients:
- One X server running on the machine in front of you. (Think of an X server as a virtual video monitor with thousands of input jacks and the remote applications as different video sources)
- One OpenSSH server on the system you are connecting to, that must be configured to (tunnel) forward X protocol

NOTE: If you are on the Windows platform, you can either purchase a commercial X server, or use Cygwin’s excellent free X server. Again… try to imagine that the X server on the machine in front of you “virtualizes” your monitor and makes it “networkable”. When you run the X server, it opens a connection on your machine listening for traffic coming from either the network, or (on *nix) shared memory. When you run an X application, it is instructed to connect to the X server so the Xserver can display it’s output. (Just like plugging in a game console to one of your TV/Monitor’s composite inputs)

Today we will talk about running X applications remotely using OpenSSH. Normally if you run X applications remotely, your X protocol traffic is going over your network connection out in the open. This is all well and good if you can trust the network that your X traffic is travelling on. But, what if you can’t? This is where ssh and X make a pretty good team. You still run your X server on the machine in front of you like usual. But instead of instructing the remote application to connect directly to it, you use OpenSSH’s X Protocol Forwarding so that all X traffic is sent through an encrypted TCP tunnel.

SSH Server Side Preparations
You will need to edit your sshd_config file which controls how your SSH server works. You would make these changes on the machine you are connecting to. First, find your sshd_config file. Typically, it’s in /usr/local/etc if you compiled the OpenSSH suite yourself or if the package maintainer went with defaults. In other cases it could be in /etc/openssh or /usr/local/etc/openssh. To verify for your distribution, you can run the ‘find’ command:

find /usr -name sshd_config

Once you’ve located it, make sure you add these lines to it for TCP and X Forwarding:

AllowTcpForwarding yes
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes

You will then need to restart your ssh server so that it reads the new configuration.

SSH Client Side Preparations:

A quick aside about the .ssh directory in your home directory. Not everyone is familiar with the purpose of this directory, but to simplify using OpenSSH, it’s another essential tool. If you don’t already have one, create a text file in ~/.ssh called ‘config’ and add something like this to it, and save it:

host work
hostname 192.168.1.10
User george

Now, if I type ’ssh work’ it will automatically try to connect to 192.168.1.10 passing ‘george’ as the user name. Obviously, you will need to adjust this appropriately to your correct IP and username. Combine that with a shared key for passwordless connection, and your life with OpenSSH becomes a lot easier. Now back to the task at hand.

Testing from the workstation in front of you:
Now we can test and see if we can forward a simple X app over the ssh tunnel. Assume the existence of ‘~/.ssh/config’ file and connection profile I create above. Also assume that the remote system has the simple X application ‘xeyes’ in /usr/bin:

ssh -X work “/usr/bin/xeyes”

We should now be seeing the familiar googly eyes peeking at us. All the application execution is happening remotely, but displaying in front of us and it’s coming over an encrypted ssh tunnel to boot!

Add X Forwarding to Your ~/.ssh/config File:
Instead of having to type ’ssh -X work [some app]‘, you can instead enable X forwarding from your ~/.ssh/config profiles. For each connection profile you create, you can add:

ForwardX11 yes

This means that all you would have to do to run a remote X app is either log into a shell using that profile and type the name of the X app you want to use, or… You can create a script to run the app using ssh and make an icon for it on your Windows Quick Launch bar or Gnome Panel. A sample script in Linux would be:

#!/bin/bash

ssh work “/usr/local/bin/gimp”

Pretty simple, huh? In Windows, you can write a CMD file using about the same syntax:

C:\cygwin\usr\local\bin\ssh work -F “C:\Documents and Settings\george\.ssh\config” “/usr/local/bin/gimp”

You can argue that this is a form of “application publishing” to use a friendly term. But it’s really a way of exploiting the features of X in a more secure way and without needing to open anything other than port 22 for OpenSSH. Once everything is configured, it works pretty seamlessly as well.

Compression:
This X traffic can take up a good deal of bandwidth since it is quite chatty back and forth, and I personally don’t use it unless I have a fast connection (DSL 1.5M or better). In the past I used to prefer ‘vnc’ over ssh for most instances and these days I use Nomachine NX protocol (which I will discuss at a later date) for remote desktop access. However, there is something you can do which might help out a little in terms of speed with X if you really don’t have any other options. You can compress your ssh traffic. Just add these lines to each host profile in your ~/.ssh/config file:

Compression yes
CompressionLevel 9

You can set your CompressionLevel to anything between 1 and 9 with ‘1′ being the fastest but worst compression, and 9 being slow but better. There is a slight improvement in X application performance. This compression applies across the board to any ssh traffic for that connection profile though, so it’s handy to add it to your slower connections.

Final Words:
Again, I don’t pretend to know everything there is about ssh or X and I am sure there are other ways that this can be done better. If you know of any, I am hoping that some of you will have more suggestions that readers here can share.  Please comment.


Any of us who are familiar with OpenSSH assume that everyone who wants to use it knows how to. But every day, there is some new thing about it that we learn and there are plenty of people who know little about it. I am hoping to share some of what I consider to be essential knowledge about the OpenSSH client as well as dispel some misunderstandings. It is quite a powerful tool when you really delve deeply into it.

Q1. Isn’t OpenSSH just an encrypted telnet program?
A1. Short answer, no. The more complete answer is that it’s a suite of programs that provide:
-remote shell access (ie. like telnet)
-remote execution of programs (both text and gui)
-remote data pipes for programs that use standard in/out
-data compression
-TCP tunneling
-ftp-like file transfers
-rcp-like file copying
-public key encryption (of all data passed between client and server)/authentication (no need for passwords)
-GUI login prompt for remote execution of X applications with ‘gnome-ssh-askpass’
-More recently, VPN functionality by way of the Linux tun/tap virtual network device driver

And I’m sure there’s more… I’m kind of an intermediate user of OpenSSH.

Q2. Setting up tunnels is a pain. What’s up with this Local/Remote Forward stuff?
A2. Actually, ‘man ssh_config’ is your friend. If you become familiar with the ~/.ssh/config file, you will find yourself not needing to type much to make connections with OpenSSH. Nearly every command line option for the ssh client can be controlled in this file. For example, I’ve set up some parameters in my ~/.ssh/config file and called the profile “home”. Now I just type: ’ssh home’ and I’m in with all the client options in place. The following is an example of some useful things to put in your ~/.ssh/config file:

-Assume that I am connecting from internally at my home (192.168.1.0)
-192.168.1.5 is my web server at home
-192.168.1.2 is my workstation at home
-We’ll pretend my workstation at work has TCP port 5022 accessible on the internet for it’s OpenSSH server and that it’s internet routeable address is 192.168.50.1 (which we know is in private address space. Just pretend…)

Example ‘~/.ssh/config’:


# My ‘webserver’ connection profile. All I
# need to do to ssh into the web server now
# is type ’ssh webserver’. I am automatically
# prompted for the password for ‘george’.
host webserver
hostname 192.168.1.5
User george


# My ‘work’ connection profile with non
# standard port for ssh (5022).
#
# I’ve also included one LocalForward line to
# forward port 80 from a web server at work to
# port 4080 on my workstation in front of me.
# So… if I connect with ’ssh work’ and log in,
# and point my browser here at home to
# 127.0.0.1:4080, I see the internal web site
# at work here at home.
#
# The RemoteForward line works in reverse. It
# sends specified TCP ports from my workstation
# in front of me at home to my workstation at work. In
# this case, OpenSSH is pushing port 5900 (vnc)
# from 192.168.1.2 (here at home) to my
# workstation at work. If I leave this connection
# up and go to work. I can run ‘vncviewer
# 127.0.0.1:0′ in my office at work and log into my workstation at home
# with vnc if needed.
host work
hostname 192.168.50.1
port 5022
User george
LocalForward 4080:privateintranet.employersnet.com:80
RemoteForward 5900:192.168.1.2:5900

Q3. Yeah… but I use Windows and I don’t have time to mess with Linux. So how does this help me?

A3. The best way to get OpenSSH (client and server) going under Windows and enjoy all the benefits is to install and use Cygwin (click here to download the installer). It’s pretty straightforward if you aren’t the sort who is afraid to get into a little *nix command line on your Windows box. You have the option of either installing a full Cygwin environment or just installing the needed base components, OpenSSL, OpenSSH, and some admin utilities to run OpenSSH as a Windows service. There are a few sets of instructions on the Internet to get you started, this being one of the better ones. At one point there was a Windows installer for OpenSSH, but it is no longer maintained and so is too out of date to consider at this point. The recommended path is Cygwin. Finally, if you’re the kind of person who uses Windows and Linux and you compile stuff from source, that is also an option with Cygwin. Just make sure you have the GNU tool chain installed in Cygwin.

Q4. Why are the docs about OpenSSH on the net so hard to understand?
A4. It took me a good deal of digging to try and find some useful info for tunneling and Public Key info for OpenSSH when I was first starting out. So, yes, the documentation could use some major improvements for people who are a little less technically inclined. To be honest, I think a nice GUI framework around OpenSSH would go a long way to getting more people to use it. There is PuTTY for Windows and it can be made to work in the context of what we’re talking about here, but it has the problem of just presenting itself as a telnet-like client and immediately turning people off who don’t need “telnet”. For example, “I never use shells or command line, why would I need an ssh telnet client”? Trying to convey the fact that telnet and OpenSSH are not the same thing, is difficult. If there was just an OpenSSH standalone “Tunneling” GUI app, I think more interest in OpenSSH would grow on the Windows side. Still, when it comes to the basics of OpenSSH on Unix, the man pages are currently the best resource. The best places to look are:


‘man sshd_config’ – Tweak your ssh server to do exactly what you need
‘man ssh_config’ – Find out what else you can do with ~/.ssh/config to minimize your command strings
‘man scp’ – Learn how to copy files AND directories using ’scp’
‘man sftp’ – A command line FTP-like interface for putting and getting files viw OpenSSH (I used to use this all the time, but have since moved to ’scp’.)
‘man ssh’ – Check out the less frequently used options.

Q5. Someone mentioned that I can use ’ssh’ in combination with ‘tar’ or ‘rsync‘ for remote backup. Is that true?
A5. More or less, depending on what you consider a useful backup. I’ve used ssh and tar for “imaging” Linux boxes. It works well, but has expected limitations. A quick example of using ssh, tar, and gzip to “image” BoxA to BoxB. Assume that we have set up ~/.ssh/config to include all needed info for username and hostname:

Backing up ‘/’ on BoxA to BoxB, intitiated from BoxB:
ssh BoxA “tar -cf – / –exclude=/proc/* –exclude=/var/tmp/*” | gzip -c > /home/admin/images/BoxA.tar.gz

Restoring ‘/’ to BoxC from the archive on BoxB, intiated from BoxC:
ssh BoxB “gunzip -c /home/admin/images/BoxA.tar.gz” | (cd / ; tar -xvf -)

You can also use the excellent ‘rsync’ command to synchronize two directories on two different machines and with the ‘-e ssh’ option tunnel it all through OpenSSH for encryption. I use this method to backup the family photos from the file server at home to a file server at my parent’s house on a nightly basis.

Using ‘rsync’:
rsync -auvlxHS -e “ssh george@remoteserver:/remotedirectory /localdirectory
Remember ‘man rsync’ is your friend…

Q6. Did you say something about VPN?

A6. Yes. With a big note saying that I’ve not tried this yet: As of version 4.3 of OpenSSH, when used in combination with the Linux kernel (or the Windows TAP/TUN Win-32 driver) tap/tun module allows for real IP tunnels between both endpoints. These tunnels are capable of relaying TCP, UDP, ICMP traffic. This is not port forwarding as discussed above, but instead true VPN. I will likely post some more info once I do try it out. I’ve been using the OpenSSL based OpenVPN for my VPN needs and have been fairly happy with it. However, it might be nice to standardize on OpenSSH for VPN.

Well… that’s it for now. There’s more to come. I know that there are people more equipped to discuss this than I am, but I am hoping to attract the curious who haven’t had the time or energy to try. I encourage anyone who is curious to give it a try no matter what platform you’re on. In the long run it’s quite a valuable tool.


The Scenario:
You’re on a system with a much older version of NIX on it than you should be using. In the middle of doing your work, you find the certain utilities are missing, or are old enough that they have some key features missing. What do you do? Tell your boss the system sucks, and go buy a new one? Wipe the box and all the important data you’re supposed to be working on with a new installation of some free *nix? Throw up your hands and do everything manually? Give up?

The Solution:
In most cases, if you’re working on a box like this, it’s likely that your dealing with text output. That was what I was dealing with and the version of egrep was too old to parse the following:

egrep '(cn:|inetUserStatus:|mailUserStatus:)'

So what did I do? I took advantage of the newer egrep on my Linux workstation using ssh:

/opt/iplanet/server5/shared/bin/ldapsearch -D "cn=MailAdmin" -w t1ck3tsplz -b "o=child.org, o=parent.org" "uid=*" | ssh george@10.0.1.25 "egrep '(cn:|inetUserStatus:|mailUserStatus:)' -" | less

The end result is that the text stream from the old *nix box is sent via ssh to my workstation where I run the egrep using STDIO. Works like a charm! And it will work with any command that outputs text that you need to process somehow. Hope this helps someone else in a similar bind.


Enter Commercial Xen (VirtualIron):

Last year, I was planning a migration away from a mail system I wasn’t happy with to the Zimbra groupware system (which looks and works great). However, I really wanted nearly unstoppable uptime even in the event of hardware failure. I knew that Xen’s live migration capability would offer me that. Using the freely distributable open source software I ran into several issues during implementation. VirtualIron was the product that finally came in to solve all of those problems.

When I first set out to virtualize Zimbra, I tried installing it on a RedHat Enterprise Linux (RHEL) paravirtualized machine running on top of Gentoo Linux with a Xen kernel. As soon as I tried to install it, Zimbra complained that I needed to have NPTL (native POSIX thread library) support in the kernel. This was not possible with Xen in paravirtualized mode. The only options I had were to run RHEL on bare metal, which would not afford me the unstoppable uptime, or to run it in a Xen HVM (full virtualization) environment. For my second attempt, I chose the second route. I was unaware of these issues when I gave Xen a second try. I mentioned in part 1 that HVM has network and disk I/O issues, but they are resolved by VirtualIron’s VSTools software.

False Starts:

I got a system that I could test with and set up a TEST Zimbra environment with CentOS 5 as Domain0 and RHEL 5 as an HVM domain. Fortunately, I ran into problems pretty quickly while testing. The first big one, was that fully virtualized Xen HVM domains CANNOT be live migrated or paused. The second issue was, of course, the bottleneck in the RAM utilization on Domain0. If your HVM domain’s disk and network I/O is very high for an application, you’ll likely wipe out all the RAM in the Domain0 and performance will suffer as your disk and network I/O attempt to work via swapping or experience long wait states. The third point, which wasn’t really an issue but more a concern from experience, was that this is whan I found out that Xen’s fully virtualized environment was really a specialized QEMU process. This was a bit worrying to me given that I wasn’t too impressed with QEMU’s performance at that point in time. Specifically because of the disk and network I/O issues mentioned above. But, I didn’t yet know the details as to the cause.

So after seeing poor disk and network performance, I did more research and more digging around for other possible approaches. I briefly considered the OpenVZ project which doesn’t really virtualize, but is more akin to chroot. While it’s quite useful and can do many of the same things that Xen can do, it’s a completely different approach and one that I wasn’t fully comfortable with. Specifically because all virtual environments are running under one Linux kernel. Then I found a blog entry comparing virtualization techniques and noted a reference to VirtualIron’s Xen base product that explained the limitations of Xen 3.x’s HVM domains and how VirtualIron worked around them. Knowing what I knew now, I recommended that we purchase VirtualIron for production. For my third attempt, we bought VirtualIron’s version of Xen which turned out to be very nice. I was expecting “your grandfather’s virtualization techniques”, but I was completely mistaken as I would find out later.

Learning VirtualIron:

One of VirtualIron’s big points is that they don’t use paravirtualization at all. This isn’t really a good or bad thing, it’s just their way of approaching virtualization. They have also been contributing back to the Xen project, so good on them! Instead, they chose to focus on the special version of QEMU included with Xen to bring it up to speed for their product. So they made sure it could do live migration! They also worked around the disk and net I/O issues by creating custom drivers and management software (The aforementioned VSTools) to be installed in the guest after you have the OS running. This limits your choice of OS to run in HVM domains unless you’re willing to build your own VSTools from their recently opened source. They currently support Windows guests up to Windows 2003 Server, and many of the most common “big name” Linux distros. Previously, VirtualIron was using a different proprietary virtualization technology which gave them a chance to develop their management tools into the robust system they are today. They moved over to Xen later but kept their management methodology which works quite nicely with Xen’s best features.

For our solution, we got two big servers for physically hosting our VMs. HP servers with 32 gigs of RAM each, and two dual-core Xeon 64-bit CPUs each. They also have fiber channel interfaces that connect to an HP SAN back-end where we store our HVM domain images. Original I had assumed that I would install VirtualIron on each of these boxes over top of a normal Linux distro, just as I did with Xen kernel installations or any other typical virtualization technology. I did just that and was lost for a bit since this was the completely wrong approach. All it seemed to do was install a DHCP server, a TFTP server, and a Java App server (Jetty if you’re curious) and no Xen kernel. Where was the Xen kernel for the systems to boot into? Digging into their online documentation provided me with a diagram of the architecture of VirtualIron which clarified things considerably. The Java based management interface for VirtualIron contains a “walk through” set up document in a pane on the right hand side of the interface. THAT is where I finally learned and understood the actual architecture and layout. My assumptions were originally completely incorrect. I should have read the manual first!

To use VirtualIron Enterprise (we didn’t go with their Single Server product which DOES work like VMWare and others) you need at least one “management server” and one “managed node”. The management server can be one of a few supported Linux distros, or Windows. The fact that the manager could be a Windows box really confused me at first, because I couldn’t understand how they would get a Xen kernel installed under an already existing Windows installation. (Yes Virtual Iron’s manager can easily be installed on a Windows server to manage the raw hardware nodes.) Again, I was still doubtful about their approach and so was wrong in that line of thinking. But, once I understood the architecture, I was both in awe and very eager to see this thing work. So I proceeded…

The VirtualIron Way:

In my case, I have two managed nodes (those monster servers with 32 gigs each) and one manager (a Xeon dual CPU 32-bit system with 2 gigs of RAM and dual NICs). The manager is running CentOS 4.5, which is supported by VirtualIron as a host for the Enterprise manager. Once I had that installed and had the management network up (you need a separate LAN dedicated to the manager and each node that you can consider “out of band”), I set one of my managed nodes to do a PXE network boot off of the manager. That’s correct, you DON’T need to install a single thing on the managed nodes, they boot via the network from Xen images stored on the manager. It’s all diskless booting via the NICs. The TFTP server and the DHCP server give this box an IP address, and point it to a pre-configured Xen microkernel boot image. Their boot image is a Xen hypervisor with a very stripped down Suse Linux Enterprise 10 (SLES10) on it. So stripped down that the managed nodes can run headless as there is ZERO interaction on those boxes other than the power button.

Once the managed node loads it’s boot from the network, it shows up in the Java management interface and you’re ready to create VMs and assign them RAM, CPU, network and storage. (NOTE: You need to check their hardware compatibility list to see if your intended server hardware is supported) In our case, the SLES10 image has drivers for our Emulex LightPulse fiberchannel HBAs, so LUNs presented by the SAN are fully accessible from within the VirtualIron manager (storage connects to the virtualization nodes, not the manager). Once VirtualIron was up, I was off and running installing RHEL 4.5 for my Zimbra installation and the special drivers and software to improve disk and network performance as well as enable live migration for HVM domains. Performance of the virtual machine was definitely very impressive. Also, keep in mind that the managed nodes that host your HVM domains don’t need to have anything installed on them in any way at all. No OS, no kernel, no boot loader, absolutely nothing. This makes the nodes essentially hot swapable as long as you keep your VM utilization across both low enough that one node can host all VMs. Beyond that, not only do they run headless, but you don’t need ANY storage in them at all if you don’t want it and have a SAN or other remote storage like iSCSI. All VM configuration resides on the managing server. So that’s the system you want backed up reliably. But there is literally nothing on the virtualization nodes at all.

Giving it a Go:

After I got it all up and running and had a Zimbra TEST instance running with VSTools installed, it was time to try a live migration. I opened up the groupware web based client and started some work in it to mimic a standard user. Then I opened the VirtualIron manager and located the Zimbra TEST instance in the Virtual Machines tree. I clicked on it and dragged it to the other physical host to initiate the migration. Then I went back and started working in my Zimbra session while I watched the migration proceed. At this point, the manager tells Xen on the source node to dump the contents of the memory for the domain that will be moved. That memory state is copied to the destination node (the other physical host) and then a final sync is done before the copy is brought up to a live state on the destination and original domain on the source is extinguished.

As this was happening, my Zimbra client continued to function completely normally. An end user wouldn’t notice a thing. For extra points, I actually shut down the physical host that originally contained the migrated domain. Absolutely no effect. It was like the HVM domain was never shutdown or moved. I’ve since made use of the live migration feature for a variety of situtions where I didn’t want down time for services but needed to make a change to hardware. After almost a year of using VirtualIron, I’m very satisfied with it from a performance perspective. VirtualIron is quite powerful and relatively inexpensive compared to other high end solutions. It can bring the power of Xen virtualization to anyone who wants it, even if they’ve never touched Linux at all. I find that to be quite an amazing thing considering Xen’s complexity. So if you have a need to run an unmodified OS (Windows or a supported Linux distribution) on Xen, I would highly recommend the VirtualIron product. However, if you are like I am at home, and run Linux nearly exclusively on the server side, consider the open source version of Xen itself using paravirtualization. Paravirtualization does not have the disk and network performance issues that HVM does.


A Little Xen:

This blog entry is going to be a bit long. You’ve been warned. :) I’ve been using virtualization on x86 hardware since 1997/98 when I got my first copy of VMWare Workstation. I used PC emulators before on other platforms, but I could tell this was different. The performance was infinitely better than any emulators I’d ever used and I was suitably impressed. Since that time, I’ve also looked at other virtualization programs like Virtual PC on the Mac and later Windows, QEMU on Linux and Windows hosts, and within the past three years, I’ve solidly focused my attention on Xen. This is largely because of their completely unique approach to virtualization and the abilities Xen offers. Abilities that nothing other than VMWare’s high end product, ESX, offers.

The thing that led me to Xen was that renewing the VMWare Workstation license was becoming prohibitive and unjustifiable for hobbyist use. (That’s changed since then but I’m into Xen now.) However, I still needed something to virtualize Windows XP since I no longer had to run it on bare metal for my needs. I’m not a gamer, and audio and video production tools for Linux are currently considerably better than they used to be, but I still need occasional access to “Windows only” stuff. At this time (2004) I was using QEMU with it’s accelerator but I wanted something better. While looking around on the net for other free virtualization systems I found the Xen project. I decided to install it on my Fedora Core 2 box and see what it would do. It didn’t start off that well because I got the Xen hypervisor (microkernel) to boot, but the console would then stop giving me any output or way to interact. I assumed it wasn’t working until I pinged the IP of my Fedora Core 2 box. It responded! I could ssh in! Weird. It was running, but no output on the screen. What else can I say other than I was intrigued.

Class Time:

Before I move forward, some background on Xen and a little vocabulary lesson is in order so I can communicate the Xen difference a little better. Most virtualization software usually requires that you install a “Host” operating system and then once that’s up and running, you install the virtualization platform itself and finally you install one or more “Guests”. While this works, quite well I might add, there is generally a performance hit because of the intermediate host OS. At the core of the virtualization process, there is a mixture of approaches to get hardware access from the guest redirected to either real hardware on the host or virtual hardware in software. Again, this can cause some reduction in performance as well as impose some limitations on hardware access itself. The way most virtualization techniques try to work around this is by translating the guest OS calls from a higher level of execution (ring 1, where applications usually run) to the host OS’s lower level of execution (ring 0, where the OS usually runs). This is a gross oversimplification by the way, but it will work to generally explain what’s going on.

The main difference with Xen vs. other virtualization approaches back then was that Xen attempts to create a virtual replacement for the x86 CPU. Instead of booting a host OS and then running Xen, you instead boot the Xen microkernel which then loads the first virtual machine. The Xen microkernel is very small and layers on top of the CPU, RAM, storage, network and peripheral buses to manage access for all virtual machines. So let’s get to our Xen definitions:

Paravirtualization: A system for managing access to the hardware resources of a computer system. The microkernel sits between the virtual machines and the hardware. In order for this to work, each virtual machine’s OS kernel needs to be compiled for the Xen virtual CPU target. The main benefit of this approach is that you can get almost 100% bare metal speed for all the virtual machines because they are essentially sharing the bare metal. The Xen microkernel creates something akin to a “ring -1″ and the virtual machines can then run in ring 0. (Obviously this won’t work for Windows, but I’ll get to that later) Another nice benefit is that you can isolate and reserve physical resources like PCI and USB devices and assign them to different virtual machines. For example, a PCI video capture card can be assigned to one virtual machine, while a second one could be assigned to another.

Domain: In Xen, there is no such thing as a guest OS since there is no OS that the virtual machines are running on top of at all. Instead the virtual machines are called domains. There is a required primary domain called Domain0 that you use for managing your other domains, but it’s not a guest.

PV Domain: These are domains that are paravirtualized. Currently you can only paravirtualize OSes that you can recompile the kernel on. That limits PV domains to Linux and the BSDs. But PV domains are the most powerful mode of operation, so worth exploring if you use one of these platforms.

HVM Domain: When Xen 3.x came out they introduced new support for hardware assisted virtualization using Intel and AMD’s comparable virtualization technologies. To do this, they incorporated a modified version of QEMU into Xen. This allows you to run OSes like Windows or older Linux distros that don’t have a 2.6 series Linux kernel in Xen. You will need a newer CPU that supports Intel Vanderpool or AMD Pacifica technologies though. There is a drawback to HVM regarding steadily decreaing disk and network I/O performance.

Onward:

It took me a little extra work, but I eventually figured my way around Xen and dedicated a box to using it. I then created paravirtualized Linux images and saw that the performance of my virtual machines was nearly 98% of running on the bare metal. It was really amazing, especially on such old hardware. My only limitation on the test box was RAM. If I had more RAM I could run many VMs on this old box. I got around to that and ended up with an old P II era celeron 400 with 384 Megs of RAM running three VMs very very well:

Domain0 (the management domain): Doing DHCP and NTP for my network. You typically shouldn’t have this domain doing much other than managing your VMs.
Domain1 (the “External” home server): Offering up three of my lame web sites, an SMTP smarhost, VPN services, external DNS for my domains
Domain2 (the “Internal” home server): Offering up MySQL for my dbmail installation, dbmail itself, postfix SMTP for internal use, internal DNS

The box is great for being so old and the virtualization really adds an extra layer of security that is unsurpassed (not perfect though). In fact, the Xen project’s technology was so impressive that MS itself is using the technology for their upcoming hypervisor in Longhorn called HyperV.

Enter HVM Domains:

With the introduction of Xen 3.x I wanted to try HVM domains. This was not possible until the first CPUs from Intel and AMD were released with hardware virtualization support. In August 2006 I bought a cheap Athlon 64 HP box at Best Buy and packed it with 4 gigs of RAM. I installed Xen 3.0 and either migrated from the old Celeron or set up new VMs on this box. There really isn’t much of a limit other than RAM and storage as far as I can tell. Right now, this box is running a Gentoo build VM, an Asterisk PBX VM, two domains migrated from the old Celeron and one more VM migrated from QEMU. Until December 2007 I also had a Windows XP VM (which I’ve actually moved back to QEMU for different reasons) running on this box as well.

All of the domains perform well and there is a mix of PV and HVM domains. The only issue I’ve seen on busy HVM domains is a gradual steady decrease in performance with network and disk I/O. This is something that I’ve seen others note was well. The simplest explanation I’ve found is that the virtual NIC and IDE interfaces are constructs in Domain0’s RAM. The busier the I/O, the more likely it is that the RAM will top out and you’ll get a performance hit due to swapping or waits (until RAM is free). Thus my sparing use of HVM with the free version of Xen.

What Makes Xen Really Different Though?

I thought you’d never ask. What else is there that sets Xen apart from the others (VMWare ESX excepted)? One feature is called “live migration”. If you have two identically configured physical hosts running the Xen microkernel, centralized storage on a third box for your VM images, and on the same network, you can actually move the VM between the two physical boxes with no downtime at all. All services continue to run, users don’t get bumped off, not even a slight pause of any kind at all. If you make sure that you don’t go beyond a reasonable amount of resource usage, on both physical hosts, this would allow you to move all the VMs from one physical box to another for actual hardware downtime/maintenance and no loss of service. How cool is that? For Linux domains, you can also change the amount of RAM allocated to a domain (as long as you’re running PV domains) while the system is live. Expand or shrink the RAM and then move it to another VM as needed. I’ve done this on the old Celeron a few times and it really is quite amazing.

If you use PV domains, the box can run headless and you can access the VM consoles with the Xen ‘xm’ management tool via an ssh session. All of the Xen management tools are CLI based stuff, so ideal for the CLI fans out there. Newer version of Fedora and CentOS come with Xen built in as an option, but I had a little trouble using these since they try to layer slightly immature GUI tools on top of Xen as well as change the way Xen’s VM configuration files work. If you run HVM domains, then you have the option of using VNC or SDL for your VM’s console display. VNC allows for network connection to the console, SDL does not. This is inherited from QEMU, so if you are familiar with QEMU it’s identical.

Is Anyone Actually Using Xen in Real Life?

Yes. As Xen has matured it spawned two commercial entities who made use of the technology. The first and most notable is Xensource. This company was started by one of the original founding members of the Xen project. They were acquired by Citrix in 2007. They support paravirtualization and provide special drivers for Windows that allow you to paravirtualize the OS without needing to rebuild the kernel. At least that’s my understanding. The other product is VirtualIron which is what I’m more familiar with in my work environment. They decided to go with HVM domains instead and offer special drivers to optimize the disk and network I/O performance. As a result they only formally support Windows and a small set of Linux distributions. They recently released the source for their VSTools package (drivers and utilities) so one can build support for unsupported distros but this is not a trivial task to complete. I’m working on building VSTools for Gentoo right now and it’s leading me down the path of building a custom initramfs. Fun for me, but not for most. I’ll be posting my experience with VirtualIron in a production setting in my next entry later today.


A few months ago I was looking around for some info on the GNU Image Manipulation Program (aka GIMP) and I discovered this product called “Aviary“. It’s a suite of web based applications built on Flash. After exploring the site a bit I discovered that the Aviary is actually a pretty large quite of applications with everything from image editing, to audio and video editing as well as desktop publishing and even marketing. My interest being rather high, I signed up. Last week I got my invitation to set up an account at the Aviary. It looks quite interesting.

They granted me access to Phoenix (their image editor) and Peacock (their algorithm based pattern generator. I suspect that only these applications were given to me since they are likely to be of general interest. Perhaps access to the other applications in the suite will happen if I make good use of the apps I have been granted access to, I’m not sure.

Phoenix is pretty basic. It contains the usual subset of generic tools that you would find in Photoshop. You can either start from scratch, or you can upload an image to work on. At the moment the file formats you can import or export are somewhat limited (png, tiff, jpg, gif) but this is a beta so it shouldn’t be a surprise. (I also want to mention that they use their own proprietary format for storing work on their side. The .EGG file format. .EGG appears to be something that all the apps can use.) Phoenix can do layers, the usual filters like blur and emboss, color fills, gradient fills and a whole lot more. It looks a bit spare compared to Photoshop or GIMP but I think that’s partially because of the way they designed the UI. Performance-wise, Phoenix is decent. I’ve noticed some lagginess when using the brush tool, but this may be a factor of the Flash plugin more than Phoenix. I’ll note that I am using Aviary in Firefox 2.x with Flash 9 on Gentoo Linux. I don’t know what the experience on Windows with IE is like, but it may be slightly better since Flash seems a bit more mature in Windows. I won’t talk about Phoenix more at this point since it’s really pretty much on par with the main features of Photoshop and GIMP and will obviously grow.

Peacock, on the other hand, is quite interesting. If anyone is familiar with the use or concepts behind analog synthesizer modules, Peacock does the same thing, only for images instead of sound. You start off with a tab bar down the left side of your screen. You have the option to choose from Generators, Effects, Controllers, and Resources. The generators are functions that create a basic shape, image or pattern. They have a few different kinds of noise, a 3D warp grid, a spirograph styled pattern, and a few others. You drag and drop a generator onto your workspace and it automatically connects to the canvas object with a “cable” (I’m making up some of these terms as I’ve not seen any documentation that explains what their terminology is).

Depending on which object you have currently selected, there are a range of settings you can adjust in the right hand pane. For the generator objects these are usually setting that determine color, shape, size, position, output level, etc… For the canvas object, you only have the ability to adjust the size (it defaults to 800×600 pixels). Next you have the effect objects. There are a significantly higher number of effect than generator objects. Some examples are glow, drop shadow, and seamless. Like the other objects, you drag and drop a filter onto the workspace. If you put it on the “cable” connecting the generator to the canvas it will automatically insert itself into the connection path. Once it’s attached and you select the effect object, you can again adjust parameters in the right hand pane to control the effect.

The controller objects are used to do things like blending (mixing two or more outputs from a chain of generator and effect objects), splitting, combining or splitting color channels and a few other functions. When using the blending object, you have ten inputs. You just “plug in” the output of some other object(s) into each input. Then with the blend object selected, you can adjust the relationship between the objects. Generally the relationship is comprised of the blend mode (like GIMP’s Normal, Add, Subtract, Multiply, etc…) and it’s intensity value. The end results can be very interesting.

Finally, there is the resources tab. Here you can pull in either images that you’ve uploaded to your “MyAviary” storage area or .EGG files from previously saved work in Aviary. At this point I did run into something odd. The imported resource didn’t show up. It was something I created in Peacock itself so maybe that was the issue. I will likely wind up explaining what I was doing to see if it might be a bug. The people behind Aviary want bug reports and feature requests since they are in beta. I suspect that if I participate and make good use of the applications, I may be granted access to some of the other applications in their suite as I’m interested in their video and audio editing applications. That’s all supposition on my part though.

Some of the nicer features I’ve found in my limited time exploring the applications are:

1. Tags in the file save dialog box

2. File open dialogs allow you to pull from: your local machine, a URL, your MyAviary storage space, Flickr, Picassa and Facebook

3. Although it takes getting used to initially, the workspace approach is not your typical MDI (multiple document interface). You have a solid workspace for your work to be done on and the tools aren’t palettes but are instead boxes that you can expand out to access variants of a particular tool type or you can compress back into just a small box.

Overall, I’m pretty impressed with Aviary. But at the moment I have no idea what their business model is or what will happen once the beta is over. Having centralized applications and data that are cross platform and that are accessible from nearly anywhere is very nice, but only if your data and the applications will still be there essentially forever. Technically, I also wonder about the limitations of writing the application in Flash. I noticed that running filters on larger photos is a lot slower than running them in GIMP locally. I suspect that this is a function of Flash not necessarily being optimized for the heavy number crunching that the filters do. However, it could also be due to the Linux Flash plugin. So I’ll probably reserve my final judgment until I see their full product. For now, it’s all a very nice diversion.


Starting off with a large dose of honesty:

At the behest of a web-head friend of mine, I’ve started this experiment in Web 2.0-ism. This blog is meant to be the real me. Well… within reason. There are things I can’t do or express here since this is supposed to be my professional image, so this is really the filtered me. It’s, as I said above, also meant to be an experiment for me. Can I adapt to the web? So far, it’s been something I’ve been avoiding. I’ve been trying to figure out why I’ve been avoiding it for the past few years and still haven’t come to any startling conclusions. I think part of it might be that I’ve done this all before.

Back when we had Usenet in the days when spammers didn’t even register so much as a minor blip, I did everything that people are talking about today. Sharing ideas? Check. Sharing media (music, images, video)? Check. Building communities? Check. Moving technology forward? Check. I even had a chat once with a guy from Norway who shared the same interest that I did in building a new OS from scratch that wasn’t owned by any company but was freely distributable. (I don’t think it was who you’re thinking it was.) I got a little quote to put into a college paper of mine from Stephen Hawking via one of his assistants. The vast world was at my fingertips. The virtual world was in full swing in 1988. At the time I had this idea of layering a nice virtual reality interface over the text interface and everything would be like having a Second Life. But it wasn’t to be. I turned right and the world went left.

So perhaps some of my hesitance in accepting this new Web 2.0 warm and fuzzy world is because having been through it all before, I’m not wanting to go through it all again only to have it usurped by the “Next Big Thing” ™ in another decade or so. My friend tells me that I need to be realistic and accept the lack of permanence. That’s very hard for someone like me to do. However, I have reasons for wanting to make myself more visible, and as my friend points out, not having a professional web presence these days is like not having a business card. My friend is very likely right. So here I am. Coming off of my Usenet buzz from 20 years ago and attempting to re-invent myself for the next fifteen minutes. Will it work? Who knows?

I’m All About Sharing:

Frankly, the other reason I’m here is that I have some knowledge locked inside my head that I’m always willing to share. I won’t claim to be an expert on most things because I’m not that kind of person. But, I do know that people are always telling me that I could help a lot of people by sharing what I know. That’s something that attracted me to Usenet and currently internet forums. I like to help people. I like to share information. I love to teach. And I have a wide variety of interests in both my personal and professional life. So that is what I will be focusing on here. A little of the personal, a little of the professional, but always trying to share what I know with you, the reader.

Under Construction:

My friend is probably shaking her head at the above heading. (Hi there!) But, this will probably be true for a while since there is a lot I need to learn about blogging and web-centric thinking. It just doesn’t come to me from second nature. I still function at Web 0.5 since it all still works well. But, I know I’ll need to modernize a bit and learn the new way of doing things if I want to have a decent presence. So, all I ask is that you be patient with me. This stuff is very hard for me because I’m not used to working at levels higher than shell scripts and text editors. I’m also not used to giving my information over to a third-party for handling. I’m told these are not good traits to have and be Web 2.0-ish. Keep in mind that we’re working against my nearly 40 years of experience being who I am. So, I’ll be trying to adapt and there will likely be false starts, wrong turns, and well… let’s just say lots and lots of mistakes initially. Sorry in advance.

What’s Next:

I will be trying to cull from various sources, previous knowledge that I’ve shared with others in the past few years that I think are still likely to be of use to people. Most of the topics from my professional side will focus on the Linux and Unix worlds and how I’ve made the move from Macintosh, to Windows, to *nix in a little over a decade. One thing I want to make clear is that I’m not an OS zealot. The way I see it, when people say, “[insert OS] sucks”, they really mean, “[insert OS] sucks for me”. If you’ve been working with multiple platforms for a number of years you know that there really is little difference other than the colors, where the gear shift is placed, and whether or not you have a spoiler. My personal experiences and needs have made me prefer Unix and Linux based distributions, but I don’t begrudge anyone choosing to use Windows or MacOS.

Over the next few weeks, I’m hoping to slowly build the amount of information I have presented here. Probably with a heavier focus on the professional at first, and then a slow movement into some of the personal. Some of this will probably be tempered by the fact that this site will be a representation of me for professional reasons. As a result, make no assumption that this blog will represent who I am 100%. While some may not feel that this is important, I do. Image is quite important to me and knowing that this blog won’t be a complete representation is something I want stated up front. Should I ever meet you at a conference or other professional function, I want to avoid anyone being surprised by any possible contrast to my online presence.

As far as the personal side goes, a bit of my background is that I was an aspiring musician with artistic inclinations who wound up in IT. As a result my entry into IT is somewhat unusual. I didn’t take computer science classes or business classes. Computers, for me, were tools to create original music, images and video 20 years ago and they still are that to me today. It just happens that along the way I learned how to do more on the “tech” side of these machines. Going back even farther, my background was stronger in electronics. Between the ages of six and 24, I could often be found modifying, building, designing and reconfiguring the raw hardware of electronic devices themselves. Mostly in the pursuit of better mediated reality experiences. The shift to computers changed my focus a bit, but I still pull out the soldering iron from time to time. Do not have any expectations that I am bent on world domination or winning some nebulous competition. I am merely here to share what I know and ideally improve people’s experiences with technology.

Fin:

So that about wraps it up for my first blog entry. The only other thing I think I’ll close with is that I tend to be time impoverished. As a result, there may be periods of inactivity which might be longer than I prefer. If the day ever does come that I just decide to throw in the towel, I’ll make it clear that I have. One thing that drives me nuts about blogs are the ones that are just abandoned with no explanation. I don’t intend to do that. Here’s to hoping that I can grasp this shiny medium while it lasts.