…for someone with the artistic eye and the technical ability to be able to pull off realism with computers. I think this is a great example of where things could go. Specifically look at the “Teasers” section. This is unlike any CGI work you’ve ever seen before:
This points to the fact that today’s artists are getting a better grasp of what the technology can do and the technology is becoming easier and more manageable for the artists to use as regular tools. This is less a study in engineering than it is a study in having a good aesthetic sense of reality.
Here’s a teaser:
T&S Teaser2 from Alex Roman on Vimeo.
Filed under: Architecture, Computer Technology, Main, art | Leave a Comment
I Might be Moving back to VMWare
Back in the late 90s I saw an ad for a free trial of something called VMWare workstation. Seeing that I’d been playing around with Wine and DOSEmu (Better alternatives today are DOSBox for games and FreeDOS for legacy apps) on Linux to gain access to some legacy applications, I assumed this was just another emulator more like Bochs. I downloaded the limited trial, installed it on my laptop and my jaw dropped.
Until this time, I was used to slow emulation where doing something like playing a 3D game (software accelerated) or playing a video was just not reliable or usable. But VMWare Workstation was different. It was also fairly inexpensive since they were new. After spending a few weeks with it, I was convinced that it was not emulation, but something completely different. I paid the price to own a copy and was in OS flipping nirvana.
My laptop running Redhat 7 could also run Windows 2000 on VMWare at what felt to be at least 80% of normal speed. I was able to play Unreal, watch music videos in MPEG and AVI formats and… do it all without a hiccup while doing the same things in Linux at the same time. I brought it into my then new employer’s offices to show it around. People were impressed.
As the price climbed, I couldn’t afford to keep up with newer versions and then I discovered the freebies like QEMU and Xen. Those worked nearly as well, but weren’t as slick. No matter for me because I like working closer to the metal anyway. The focus eventually became Xen (installing and building from source on top of Fedora, then Gentoo). When the time came to implement a new web based groupware system, Zimbra, the free Xen wasn’t up to the task. I did some more research and found an excellent commercial product based on Xen called Virtual Iron.
Virtual Iron really had everything right in terms of their implementation of the management. The management server could be run on Windows or a few supported flavors of Linux. It provided a central repository of ISOs to use for initially installing your VMs and a boot image that each of your x86_64 virtual machine hosts would boot off of via TFTP. The beauty of that is that you didn’t need to install anything on the virtual machine hosts at all. They just boot up with the custom Xen and XenLinux kernels and they’re ready to host whatever VMs you throw at them. Everything was there including hiccup free live migration.
This past Summer, Oracle acquired Virtual Iron and it currently looks like they plan to turn it more into a virtual desktop hosting solution to complement their Oracle VM product. At this time it is unclear to me how many of Virtual Iron’s former (very rockin’) support and engineering staff they retained. At one point it even looked like they might bury the product entirely, but a recent issue of Information Weekly indicates that there might still be life left in it. However, the time came for me to re-evalute virtualization.
Over the past few months as time has allowed, I’ve been investigating many of the options available for virtualization. Since my VMs are all Linux, that means I have a lot of options ranging from free/libre (as in speech, not beer), free of charge (but pay for support), or ungodly expensive. I also have to factor in support from HP for the virtualization platform which narrows the field a little. So right now it looks like it’s between Novell/Suse Enterprise (can host or be virtualized with near bare metal performance on HyperV), Redhat Enterprise, Centos, Ubuntu Server 9 + KVM or Xen, and VMWare.
This is where it gets interesting. When I first went over to the Xen side of the table, VMWare ESX and GSX were the server platforms and they were extremely expensive. ESX was unique when it came out because it was the first virtualization product for the x86 platform that didn’t require a full OS running under it. It was a hypervisor. The Xen microkernel, offered essentially the same thing, only it was free if you wanted to make the minimal investment of time to build it from source. Seeing that kernel compilation is something of a breeze for me, I gave it a try and was duly impressed.
But, VMWare didn’t stop there. They have an advantage that I don’t believe Xen is yet positioned to compete with. They can over commit your server’s RAM in some very interesting ways. Of course, anyone familiar with Xen is aware of how you can “balloon” [1] RAM for your Linux “guests” [2] If you assigned only 256 megs to a guest and now you need 512, you can use the ‘xm’ utility to resize the amount of RAM that guest has. If you’re paravirtualized, which I used a lot initially, the change in the guest is immediate. As a result you can move RAM between VMs as the demands dictate.
VMWare can also do this. But the coolest thing that VMWare can do that others can’t yet is to share similar areas of RAM between multiple VMs. As they point out on their marketing site, this allows for much more efficient use of RAM. Where products that are based on Xen like Citrix or MS HyperV are limited by how much RAM is on the host machine, VMWare can run many more VMs on the same hardware. On their example server with 4 gigs of RAM, they could only run seven and six 512 meg Windows XP VMs compared to VMWare’s 20 VMs. This is because the system copies portions of the first VMs running image that are the same on successive VMs. It then keeps track of changes to maintain independent functionality.
They argue that while they might cost more you get a lot more VMs for your dollar. And they’re right. So, I’ll be eager to see this in action as I test the various VM technologies. I want the best one I can find, and VMWare is very interesting right now.
[1] The Obsessively Compulsive blog is really excellent for virtualization info. I’ve followed it on and off for a while, it never disappoints.
[2] Guest is the wrong terminology in Xen parlance, but it will suffice for this blog post.
Filed under: Computer Basics, Computer Technology, Main | Leave a Comment
GUIs: Tiled Window Managers
Well that was a long break! I’m going to give my blog another try and see if I can keep it going a bit better this time. Sorry for that loooong intermission. 
Being someone who believes in using technology to make life better, one area that’s always interested me is human/machine interaction. It’s a topic called human factors. Having read Jef Raskin’s book, The Humane Interface, I know that there are ideas that work, and ideas that just don’t. He had some very interesting ideas, many of which have slowly been making their way into the software we use every day whether installed on our computers, or via the web.
In my own personal experience, I discovered quite some time ago that one size (or flavor) does not fit all when it comes to user interfaces. One person might think that the latest Microsoft Windows 7 GUI advances are the best thing since sliced bread, while another might think they’re a cheap knock-off of the Mac OS. Another might find both of those environments to be far too wasteful in terms of system resources and workflow efficiency. And still another might feel that only the “one true OS” has the best GUI of all time. Which OS that is, is an exercise left up to the reader.
My personal experiences have allowed me to try nearly all GUIs at one point or another for extended periods of time and there are definitely, sometimes subtle, differences. I’ve used everything from the earliest GUIs on Apple hardware, to the latest in Linux land and all in between. What I’ve personally discovered is that each GUI approach has its benefits and drawbacks to certain tasks. Another discovery is that until we get some more interesting input devices, that GUIs are generally all limited to a few basic functions no matter how much eye candy and window dressing is applied.
Having used Linux almost exclusively for many years, I was largely exposed to the Gnome and KDE environments which both take aspects of the Windows and Mac OS desktops as well as inheriting some attributes from a few Unix desktops as well. The paradigm is purely windows, icons, mouse and pointer (W.I.M.P.). During this time I also tried out many other environments and found that for normal use I preferred Gnome for day to day use. I also tried Enlightenment which is just amazing, but since it’s always in alpha, it’s not very stable for day to day use and had some problems with presenting some apps properly. It also required a little more coding experience to customize than I had at the time.
About three or four years ago, I decided to give KDE a spin again, and spent about half a year with it. I found that I loved their Konsole (terminal) app above all others. But GUI-wise it was still about as clunky as Windows 9x-2000. I breifly returned to Gnome with the default Metacity window manager, then I discovered the amazing Beryl, now merged with Compiz to become Compiz-Fusion and now renamed yet again to Compiz. This environment was packed with eye candy galore, some of which was Mac inspired, but much of which was actually unique. I used that consistently at work for about two years and a few months and I loved every minute of it. Being able to easily and visually zoom in and out of my sixteen virtual desktops was about as close as things get to Jef Raskin’s zooming user interface. (Tuft’s University’s VUE mind mapping application also has a pretty decent zoom as well)
But, as usual, I got the itch to try something different and last Winter, I found a blog post at codinghorror.com about how your desktop should not be a destination. That is, your screen should never be focused on your desktop since that means you’re not actually doing anything but looking at your wallpaper and maybe a mess of icons if you’re not a minimalist like me. 
I delved a little farther into this notion and I realized that it was time for me to try a keyboard driven window manager. It was time to step away from the mouse…
The first one I tried was Xmonad. It’s a tiling window manager which automatically places applications on your screen in the most efficient layout. The general idea is to get away from having to move, place or resize windows. Instead, the first application window is mostly full screen. There are no window widgets to close/minimize/maximize because those concepts don’t exist in this environment. When you open a second application window, Xmonad splits the screen in half with the first app on the top and the second on the bottom half. You can then switch into two other modes: side by side, and full screen where you cycle between apps with a key combo.
There are also a default of nine workspaces (kind of like virtual desktops). So it’s easy to group applications together. I would use space 1 for my web mail session at work. Space 2 for another browser with work related links in it. Space three for some terminal windows for the various shell sessions to *nix and VMS boxes. Space four, for a Windows remote desktop session to a virtual machine. Space seven, for a text editor with my latest task list in it. And finally space nine for a remote NX session (like remote desktop, but for Linux) to my app server at home via OpenVPN. The other ones I skipped I’d throw various apps onto and they tend to be transient.
Spending about four or five months with it was interesting. But I wasn’t sure if it really made that much of a difference. Sometimes the tiling paradigm just didn’t work. The split screen whether vertical or horizontal didn’t provide enough space for either app and using them in full screen mode and cycling between them wasn’t very efficient. So when I just switched to Ubuntu Desktop from Gentoo on my workstation this week, I went back to Gnome with Metacity. Only to discover that the keyboard driven window manager fans were right. It’s a massive pain in the rear to have to move, place and resize windows all the time. Even with multiple virtual desktops! I could actually feel the impact on my workflow efficiency and working with Gnome, Compiz-Fusion or Windows just feels too slow.
So I promptly installed Xmonad as well as wmii. As soon as I tried wmii, I was in heaven. The key combos are similar to Xmonad so not much time to adapt. Plus they switched from the multiple virtual desktop approach to tags for application windows. Which is much more powerful. The tags appear to be the same as virtual desktops in that there are some defaults (0 through 9) which make it look like you just have ten virtual desktops to move between. But the difference is that you can use more than one tag on an application window. This then allows you to have a single application grouped with other applications under other tags so it follows you. Even better is that if you switch an app to floating mode (which is similar to what other standard GUIs do) under one tag, that only applies there. When you switch to another tag the application will change to conform to however it’s been configured for that other tag. So I’ve noticed a definite difference in using wmii and my ability to save time during the day as I work through being able to forget about bothering with window placement and resizing unless I want to do it. Note that I haven’t yet tried Awesome or Ratpoison which also allow you to ditch the mouse. And as a second note, I’ll point out that some applications still demand a mouse, so you can’t really always keep your hands on the home keys. But wmii and Xmonad are definitely the way to go for serious work.
On that last point, I’ll note that I use Compiz-Fusion on the media center at home, and standard Gnome on a few other machines. Which is really the whole reason for this blog entry. What I’ve discovered is that you can get a lot more done with a computer if you have the right UI for the job. At work, where I can have anywhere from 15-30 windows open at a time, and I need to be able to move between them quickly or group them in various ways, wmii is a clear win. But, on the laptop that I share with my family, I need something that isn’t as byzantine, so standard Gnome or Compiz-Fusion is the best choice. When I’m working on music or editing video, I need maximal screen space combined with the occasionaly need to resize a window, so wmii it is again.
As you can see, there is no one approach that works well for all tasks, but you don’t realize that until you try all the options. So if you’re on the fence and curious about different UI approaches, go ahead and explore. What you learn will be worth it. Just make sure you give yourself enough time to really figure out how you feel about the environment. For me, that’s a minimum of six months. Taking the tiime and honestly giving it your best effort might surpris you. I remember when I first saw tiling window managers about ten years ago and wondered, “why would anyone want that”? Now I know.
Filed under: Computer Basics, Computer Technology, Main, Reviews | Leave a Comment
Enso: Humanize Your Computer
I’m in the middle of reading Jef Raskin’s (started a project in the 70s called Apple Macintosh) book, The Humane Interface. It’s a terrific read that tells us all about what’s wrong with software as it stands today. Although Jef died in 2005, his son Aza is carrying on the work because it’s just too good to leave unfinished. Aza formed a company called Humanized.com and their product is called Enso (Windows only for now). It sits between the user and the system and applies the speed, power and efficiency of the command line with the elegance of the GUI. The closest thing I’ve seen to it is Firefox Ubuiquity (Also written by Aza Raskin), only this applies to the entire system. While it’s not perfectly in line with Jef Raskin’s ideals, it’s a pretty nice step to getting there. Check out the link to the demo video above to get a peek into what could be possible. Enso is a command line for the non-technical person. Computers don’t have to be unfriendly.
Filed under: Books, Computer Basics, Computer Technology, Main | Leave a Comment
Difficulty Level: Moderate
What is a Network Block Device and Why Would I Want One?
Let me start this entry out by explaining just what a block device is, in case you’re a newer Linux user and you’re not sure. I didn’t know what one was at one point and a quick explanation would have been helpful. In short, block devices are things like hard drives, flash drives, floppy disks, CD-ROMs and even DVDs. On a more complex level, they are devices which get their input/output in the form of data blocks of a certain size in bits or bytes. For the sake of this discussion, we’ll just be thinking of the devices I listed above.
The Linux kernel among it’s many modules (which can be thought of as drivers) has a particular module called ‘nbd’ which stands for Network Block Device. What this means is that you can take almost any block device and present it to the network. This differs from standard Windows file sharing, or Unix NFS in that you’re not presenting a set of files to the network, but the raw device itself. (iSCSI is another way of accomplishing the same thing with a greater degree of reliability and is accessible to Linux through the Linux iSCSI Target and Linux iSCSI Initiator projects. NOTE: Think of the target as the server, and the initiator as the client.) Although there are other ways of doing this, network block device support is still nothing to ignore. It may not be robust, but it’s still highly useful for non-mission critical tasks where the expense of a SAN is unwarranted.
I originally found out about nbd when I was first starting to work with Xen virtualization. Their project documentation suggested that a convenient way of being able to store virtual machine images on a server was to use nbd support. When I understood that this was the way to a “poor man’s SAN” since Linux software RAID and LVM volumes could be exported with nbd, I then wondered, what else could I export? I decided to experiment and find out. While it’s not perfect because of some I/O control limitations, it’s still quite handy and simple to implement.
Making NBD Work for You:
The first step towards preparing your system for NBD is determining if you need to recompile your kernel. Fortunately, most of the popular Linux distributions tend to build most of the optional support as kernel modules by default. You can think of modules as “drivers” for different functionality and hardware support. In most cases, you should have access to NBD support. The simplest way to find out is to get to a shell prompt and type: ‘modprobe nbd’. If you get no error and simply return to the prompt, then type: ‘lsmod | grep -v grep | grep nbd’. If you see a line indicating that the nbd module is loaded, your kernel has NBD support built in.
However, if you get an error with the ‘modprobe’ command about the module not existing, or you don’t get anything in return for the ‘lsmod’ command, then it’s likely you’ll need to recompile your kernel. Recompiling the kernel is beyond the scope of this blog entry, but I will link to some resources to get you started in the right direction at the end of this entry.
Three Components to NBD:
There are three components that make up the entirety of NBD for Linux. The first is the kernel module which allows the kernel to provide an interface to the device for export to/from the network. The second component is the ‘nbd-server’ application which handles the exporting of the device over TCP/IP. And finally, the third part is the ‘nbd-client’ application which imports the device on another machine and presents it as /dev/nbX where ‘X’ is a number. Depending on the distribution you use, you may be able to find a specific package to install the applications. If not, there is always the source code from the main NBD project site.
Once you have the kernel module loaded and the applications built here is all you need to do to test it and see if it will work for what you need. This is an example of exporting a raw unpartitioned IDE hard drive over TCP/IP and then importing it on a remote system:
1. On the system containing the hard drive, run the nbd-server command as follows (Syntax: nbd-server -r <tcp port> /dev/xxx):
nbd-server -r 2000 /dev/hda
2. On the remote system where you wish to import the devicem run the nbd-client command as follows (Syntax: /nbd-client <ip address of system running nbd-server> <matching tcp port number> /dev/nbd0)
nbd-cliet 192.168.1.1 2000 /dev/nbd0
You should then be able to treat /dev/nbd0 on the system where you imported the device as if it were local. Use ‘fdisk’ to partition the device, format it for use as a file system, or even a paging file. I’ve used this successfully with remote flash drives, raw hard drives, partitions on hard drives, LVM logical volumes, and even DVD drives for playing movies on devices that don’t have DVD drives.
A few extra notes:
1. Depending on the kernel version, the NBD device nodes might be /dev/nbdX or it might be /dev/nbX where ‘X’ is always a number.
2. There is a one-to-one relationship between the exported device and your chosen TCP port. That is to say, that if you use port 2000 for /dev/hda and want to export /dev/hdb simultaneously, you’ll need to increment to the next free port.
3. Before randomly chosing ports, it’s a good idea to take a look at the commonly used ports listed in /etc/services as well as run a ‘netstat -an’ to see what ports your particular systems are using.
4. The performance of the DVD export over an 802.11bg link is quite good after an initial buffer period for something like Xine.
5. It’s quite possible to use an imported NBD device as part of a mirror set if you want a pseudo instant copy on a separate machine, but it’s not highly recommended.
Recompiling the Linux kernel (which seems to be a dying art):
Filed under: Computer Technology, Main | Leave a Comment
What Is This All About?
Every pioneering, epic journey worth taking from point A to point B requires asking for directions, wrong turns, heading in the completely wrong direction, or even getting lost. After all, you’re the scout. The first person to be sent out with enough knowledge to try and find your way to the highly desired point B. Your chances of success are greatly improved if you have some important skills and tools to take along with you. With that introduction, my intended goal of sharing some of my knowledge and tools with you regarding the world of Linux distributions can begin. Of course, before I proceed I should make it clear that I’m no expert and won’t pretend to be one. I’m just a keen experimenter, and a fan of what Linux distributions have enabled me to do. With that I begin my “From Here to There” series.
I was originally going to write a blog post on using Network Block Devices in Linux, when I kept running into the same problem over and over. My hope was that I could write an article simple enough for anyone to grasp. However, each time I went back to the entry, I discovered that there was yet another thing that I take for granted, that many Linux users might not understand. Each time I attempted to remedy that situation with some general guidelines, I found that I had to cover exceptions for the multitude of distributions out there which would really make the blog entry far too long and far too complicated for my intended audience: Linux newbies, and the highly curious users of other OSes.
So I sat down for a while and tried putting my finger on how it is that I can move between the Linux distros fairly easily and get work done. I’m no computer expert. I’m not a programmer. I’m just a basic system admin who went from Mac (pre OS X) to Windows (3.1 through XP) and got involved with Linux in 1997 out of necessity. While thinking, I realized that there are many things that we computer folks take for granted about what we do normally that fall into the category of “missing knowledge” for the average user. Without that knowledge, understanding how to use a computer is next to impossible. Many people before me have written about people who simply learn how to use specific brands of software, and do not acquire the application literacy needed to be able to use any software of the same category (ie. word processors). That’s not what the “Here to There” series will be about.
Instead, it will be my intention to try and share some of the more basic skills that I take for granted, when I use computers, that others might be unaware of. Everyone who is at an intermediate level or higher, with regard to computer use, has skills like these. But we rarely if ever discuss them because to us they are second nature. So the “Here to There” series will attempt to get people from point A to point B with a clear understanding as to how and why the journey is worth it. Since my experience is largely with Linux and Free/Open software, that is where some of my focus will be. But, in all honesty and with only a few exceptions, there really is little difference between OSes and applications today. As a result, much of what I share should apply to many OSes and applications. I will be attempting to post the first in the series this week. (Correction: Life, as usual, has become fairly busy. So I will be starting the series sometime soon. The first part should appear when it’s ready…)
Filed under: Computer Basics, Computer Technology, Main | Leave a Comment
Tags: From Here to There
- One OpenSSH server on the system you are connecting to, that must be configured to (tunnel) forward X protocol
NOTE: If you are on the Windows platform, you can either purchase a commercial X server, or use Cygwin’s excellent free X server. Again… try to imagine that the X server on the machine in front of you “virtualizes” your monitor and makes it “networkable”. When you run the X server, it opens a connection on your machine listening for traffic coming from either the network, or (on *nix) shared memory. When you run an X application, it is instructed to connect to the X server so the Xserver can display it’s output. (Just like plugging in a game console to one of your TV/Monitor’s composite inputs)
Today we will talk about running X applications remotely using OpenSSH. Normally if you run X applications remotely, your X protocol traffic is going over your network connection out in the open. This is all well and good if you can trust the network that your X traffic is travelling on. But, what if you can’t? This is where ssh and X make a pretty good team. You still run your X server on the machine in front of you like usual. But instead of instructing the remote application to connect directly to it, you use OpenSSH’s X Protocol Forwarding so that all X traffic is sent through an encrypted TCP tunnel.
SSH Server Side Preparations
You will need to edit your sshd_config file which controls how your SSH server works. You would make these changes on the machine you are connecting to. First, find your sshd_config file. Typically, it’s in /usr/local/etc if you compiled the OpenSSH suite yourself or if the package maintainer went with defaults. In other cases it could be in /etc/openssh or /usr/local/etc/openssh. To verify for your distribution, you can run the ‘find’ command:
find /usr -name sshd_config
Once you’ve located it, make sure you add these lines to it for TCP and X Forwarding:
AllowTcpForwarding yes
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes
You will then need to restart your ssh server so that it reads the new configuration.
SSH Client Side Preparations:
A quick aside about the .ssh directory in your home directory. Not everyone is familiar with the purpose of this directory, but to simplify using OpenSSH, it’s another essential tool. If you don’t already have one, create a text file in ~/.ssh called ‘config’ and add something like this to it, and save it:
host work
hostname 192.168.1.10
User george
Now, if I type ’ssh work’ it will automatically try to connect to 192.168.1.10 passing ‘george’ as the user name. Obviously, you will need to adjust this appropriately to your correct IP and username. Combine that with a shared key for passwordless connection, and your life with OpenSSH becomes a lot easier. Now back to the task at hand.
Testing from the workstation in front of you:
Now we can test and see if we can forward a simple X app over the ssh tunnel. Assume the existence of ‘~/.ssh/config’ file and connection profile I create above. Also assume that the remote system has the simple X application ‘xeyes’ in /usr/bin:
ssh -X work “/usr/bin/xeyes”
We should now be seeing the familiar googly eyes peeking at us. All the application execution is happening remotely, but displaying in front of us and it’s coming over an encrypted ssh tunnel to boot!
Add X Forwarding to Your ~/.ssh/config File:
Instead of having to type ’ssh -X work [some app]‘, you can instead enable X forwarding from your ~/.ssh/config profiles. For each connection profile you create, you can add:
ForwardX11 yes
This means that all you would have to do to run a remote X app is either log into a shell using that profile and type the name of the X app you want to use, or… You can create a script to run the app using ssh and make an icon for it on your Windows Quick Launch bar or Gnome Panel. A sample script in Linux would be:
#!/bin/bash
ssh work “/usr/local/bin/gimp”
Pretty simple, huh? In Windows, you can write a CMD file using about the same syntax:
C:\cygwin\usr\local\bin\ssh work -F “C:\Documents and Settings\george\.ssh\config” “/usr/local/bin/gimp”
You can argue that this is a form of “application publishing” to use a friendly term. But it’s really a way of exploiting the features of X in a more secure way and without needing to open anything other than port 22 for OpenSSH. Once everything is configured, it works pretty seamlessly as well.
Compression:
This X traffic can take up a good deal of bandwidth since it is quite chatty back and forth, and I personally don’t use it unless I have a fast connection (DSL 1.5M or better). In the past I used to prefer ‘vnc’ over ssh for most instances and these days I use Nomachine NX protocol (which I will discuss at a later date) for remote desktop access. However, there is something you can do which might help out a little in terms of speed with X if you really don’t have any other options. You can compress your ssh traffic. Just add these lines to each host profile in your ~/.ssh/config file:
Compression yes
CompressionLevel 9
You can set your CompressionLevel to anything between 1 and 9 with ‘1′ being the fastest but worst compression, and 9 being slow but better. There is a slight improvement in X application performance. This compression applies across the board to any ssh traffic for that connection profile though, so it’s handy to add it to your slower connections.
Final Words:
Again, I don’t pretend to know everything there is about ssh or X and I am sure there are other ways that this can be done better. If you know of any, I am hoping that some of you will have more suggestions that readers here can share. Please comment.
Filed under: Computer Technology, Main | Leave a Comment
Tags: linux, openssh, ssh, X, X Window System, Xorg
Any of us who are familiar with OpenSSH assume that everyone who wants to use it knows how to. But every day, there is some new thing about it that we learn and there are plenty of people who know little about it. I am hoping to share some of what I consider to be essential knowledge about the OpenSSH client as well as dispel some misunderstandings. It is quite a powerful tool when you really delve deeply into it.
Q1. Isn’t OpenSSH just an encrypted telnet program?
A1. Short answer, no. The more complete answer is that it’s a suite of programs that provide:
-remote shell access (ie. like telnet)
-remote execution of programs (both text and gui)
-remote data pipes for programs that use standard in/out
-data compression
-TCP tunneling
-ftp-like file transfers
-rcp-like file copying
-public key encryption (of all data passed between client and server)/authentication (no need for passwords)
-GUI login prompt for remote execution of X applications with ‘gnome-ssh-askpass’
-More recently, VPN functionality by way of the Linux tun/tap virtual network device driver
And I’m sure there’s more… I’m kind of an intermediate user of OpenSSH.
Q2. Setting up tunnels is a pain. What’s up with this Local/Remote Forward stuff?
A2. Actually, ‘man ssh_config’ is your friend. If you become familiar with the ~/.ssh/config file, you will find yourself not needing to type much to make connections with OpenSSH. Nearly every command line option for the ssh client can be controlled in this file. For example, I’ve set up some parameters in my ~/.ssh/config file and called the profile “home”. Now I just type: ’ssh home’ and I’m in with all the client options in place. The following is an example of some useful things to put in your ~/.ssh/config file:
-Assume that I am connecting from internally at my home (192.168.1.0)
-192.168.1.5 is my web server at home
-192.168.1.2 is my workstation at home
-We’ll pretend my workstation at work has TCP port 5022 accessible on the internet for it’s OpenSSH server and that it’s internet routeable address is 192.168.50.1 (which we know is in private address space. Just pretend…)
Example ‘~/.ssh/config’:
# My ‘webserver’ connection profile. All I
# need to do to ssh into the web server now
# is type ’ssh webserver’. I am automatically
# prompted for the password for ‘george’.
host webserver
hostname 192.168.1.5
User george
# My ‘work’ connection profile with non
# standard port for ssh (5022).
#
# I’ve also included one LocalForward line to
# forward port 80 from a web server at work to
# port 4080 on my workstation in front of me.
# So… if I connect with ’ssh work’ and log in,
# and point my browser here at home to
# 127.0.0.1:4080, I see the internal web site
# at work here at home.
#
# The RemoteForward line works in reverse. It
# sends specified TCP ports from my workstation
# in front of me at home to my workstation at work. In
# this case, OpenSSH is pushing port 5900 (vnc)
# from 192.168.1.2 (here at home) to my
# workstation at work. If I leave this connection
# up and go to work. I can run ‘vncviewer
# 127.0.0.1:0′ in my office at work and log into my workstation at home
# with vnc if needed.
host work
hostname 192.168.50.1
port 5022
User george
LocalForward 4080:privateintranet.employersnet.com:80
RemoteForward 5900:192.168.1.2:5900
Q3. Yeah… but I use Windows and I don’t have time to mess with Linux. So how does this help me?
A3. The best way to get OpenSSH (client and server) going under Windows and enjoy all the benefits is to install and use Cygwin (click here to download the installer). It’s pretty straightforward if you aren’t the sort who is afraid to get into a little *nix command line on your Windows box. You have the option of either installing a full Cygwin environment or just installing the needed base components, OpenSSL, OpenSSH, and some admin utilities to run OpenSSH as a Windows service. There are a few sets of instructions on the Internet to get you started, this being one of the better ones. At one point there was a Windows installer for OpenSSH, but it is no longer maintained and so is too out of date to consider at this point. The recommended path is Cygwin. Finally, if you’re the kind of person who uses Windows and Linux and you compile stuff from source, that is also an option with Cygwin. Just make sure you have the GNU tool chain installed in Cygwin.
Q4. Why are the docs about OpenSSH on the net so hard to understand?
A4. It took me a good deal of digging to try and find some useful info for tunneling and Public Key info for OpenSSH when I was first starting out. So, yes, the documentation could use some major improvements for people who are a little less technically inclined. To be honest, I think a nice GUI framework around OpenSSH would go a long way to getting more people to use it. There is PuTTY for Windows and it can be made to work in the context of what we’re talking about here, but it has the problem of just presenting itself as a telnet-like client and immediately turning people off who don’t need “telnet”. For example, “I never use shells or command line, why would I need an ssh telnet client”? Trying to convey the fact that telnet and OpenSSH are not the same thing, is difficult. If there was just an OpenSSH standalone “Tunneling” GUI app, I think more interest in OpenSSH would grow on the Windows side. Still, when it comes to the basics of OpenSSH on Unix, the man pages are currently the best resource. The best places to look are:
‘man sshd_config’ – Tweak your ssh server to do exactly what you need
‘man ssh_config’ – Find out what else you can do with ~/.ssh/config to minimize your command strings
‘man scp’ – Learn how to copy files AND directories using ’scp’
‘man sftp’ – A command line FTP-like interface for putting and getting files viw OpenSSH (I used to use this all the time, but have since moved to ’scp’.)
‘man ssh’ – Check out the less frequently used options.
Q5. Someone mentioned that I can use ’ssh’ in combination with ‘tar’ or ‘rsync‘ for remote backup. Is that true?
A5. More or less, depending on what you consider a useful backup. I’ve used ssh and tar for “imaging” Linux boxes. It works well, but has expected limitations. A quick example of using ssh, tar, and gzip to “image” BoxA to BoxB. Assume that we have set up ~/.ssh/config to include all needed info for username and hostname:
Backing up ‘/’ on BoxA to BoxB, intitiated from BoxB:
ssh BoxA “tar -cf – / –exclude=/proc/* –exclude=/var/tmp/*” | gzip -c > /home/admin/images/BoxA.tar.gz
Restoring ‘/’ to BoxC from the archive on BoxB, intiated from BoxC:
ssh BoxB “gunzip -c /home/admin/images/BoxA.tar.gz” | (cd / ; tar -xvf -)
You can also use the excellent ‘rsync’ command to synchronize two directories on two different machines and with the ‘-e ssh’ option tunnel it all through OpenSSH for encryption. I use this method to backup the family photos from the file server at home to a file server at my parent’s house on a nightly basis.
Using ‘rsync’:
rsync -auvlxHS -e “ssh george@remoteserver:/remotedirectory /localdirectory
Remember ‘man rsync’ is your friend…
Q6. Did you say something about VPN?
A6. Yes. With a big note saying that I’ve not tried this yet: As of version 4.3 of OpenSSH, when used in combination with the Linux kernel (or the Windows TAP/TUN Win-32 driver) tap/tun module allows for real IP tunnels between both endpoints. These tunnels are capable of relaying TCP, UDP, ICMP traffic. This is not port forwarding as discussed above, but instead true VPN. I will likely post some more info once I do try it out. I’ve been using the OpenSSL based OpenVPN for my VPN needs and have been fairly happy with it. However, it might be nice to standardize on OpenSSH for VPN.
Well… that’s it for now. There’s more to come. I know that there are people more equipped to discuss this than I am, but I am hoping to attract the curious who haven’t had the time or energy to try. I encourage anyone who is curious to give it a try no matter what platform you’re on. In the long run it’s quite a valuable tool.
Filed under: Computer Technology, Main | 2 Comments
Tags: linux, openssh, ssh, unix
The Scenario:
You’re on a system with a much older version of NIX on it than you should be using. In the middle of doing your work, you find the certain utilities are missing, or are old enough that they have some key features missing. What do you do? Tell your boss the system sucks, and go buy a new one? Wipe the box and all the important data you’re supposed to be working on with a new installation of some free *nix? Throw up your hands and do everything manually? Give up?
The Solution:
In most cases, if you’re working on a box like this, it’s likely that your dealing with text output. That was what I was dealing with and the version of egrep was too old to parse the following:
egrep '(cn:|inetUserStatus:|mailUserStatus:)'
So what did I do? I took advantage of the newer egrep on my Linux workstation using ssh:
/opt/iplanet/server5/shared/bin/ldapsearch -D "cn=MailAdmin" -w t1ck3tsplz -b "o=child.org, o=parent.org" "uid=*" | ssh george@10.0.1.25 "egrep '(cn:|inetUserStatus:|mailUserStatus:)' -" | less
The end result is that the text stream from the old *nix box is sent via ssh to my workstation where I run the egrep using STDIO. Works like a charm! And it will work with any command that outputs text that you need to process somehow. Hope this helps someone else in a similar bind.
Filed under: Computer Technology | Leave a Comment
Tags: openssh, ssh
Enter Commercial Xen (VirtualIron):
Last year, I was planning a migration away from a mail system I wasn’t happy with to the Zimbra groupware system (which looks and works great). However, I really wanted nearly unstoppable uptime even in the event of hardware failure. I knew that Xen’s live migration capability would offer me that. Using the freely distributable open source software I ran into several issues during implementation. VirtualIron was the product that finally came in to solve all of those problems.
When I first set out to virtualize Zimbra, I tried installing it on a RedHat Enterprise Linux (RHEL) paravirtualized machine running on top of Gentoo Linux with a Xen kernel. As soon as I tried to install it, Zimbra complained that I needed to have NPTL (native POSIX thread library) support in the kernel. This was not possible with Xen in paravirtualized mode. The only options I had were to run RHEL on bare metal, which would not afford me the unstoppable uptime, or to run it in a Xen HVM (full virtualization) environment. For my second attempt, I chose the second route. I was unaware of these issues when I gave Xen a second try. I mentioned in part 1 that HVM has network and disk I/O issues, but they are resolved by VirtualIron’s VSTools software.
False Starts:
I got a system that I could test with and set up a TEST Zimbra environment with CentOS 5 as Domain0 and RHEL 5 as an HVM domain. Fortunately, I ran into problems pretty quickly while testing. The first big one, was that fully virtualized Xen HVM domains CANNOT be live migrated or paused. The second issue was, of course, the bottleneck in the RAM utilization on Domain0. If your HVM domain’s disk and network I/O is very high for an application, you’ll likely wipe out all the RAM in the Domain0 and performance will suffer as your disk and network I/O attempt to work via swapping or experience long wait states. The third point, which wasn’t really an issue but more a concern from experience, was that this is whan I found out that Xen’s fully virtualized environment was really a specialized QEMU process. This was a bit worrying to me given that I wasn’t too impressed with QEMU’s performance at that point in time. Specifically because of the disk and network I/O issues mentioned above. But, I didn’t yet know the details as to the cause.
So after seeing poor disk and network performance, I did more research and more digging around for other possible approaches. I briefly considered the OpenVZ project which doesn’t really virtualize, but is more akin to chroot. While it’s quite useful and can do many of the same things that Xen can do, it’s a completely different approach and one that I wasn’t fully comfortable with. Specifically because all virtual environments are running under one Linux kernel. Then I found a blog entry comparing virtualization techniques and noted a reference to VirtualIron’s Xen base product that explained the limitations of Xen 3.x’s HVM domains and how VirtualIron worked around them. Knowing what I knew now, I recommended that we purchase VirtualIron for production. For my third attempt, we bought VirtualIron’s version of Xen which turned out to be very nice. I was expecting “your grandfather’s virtualization techniques”, but I was completely mistaken as I would find out later.
Learning VirtualIron:
One of VirtualIron’s big points is that they don’t use paravirtualization at all. This isn’t really a good or bad thing, it’s just their way of approaching virtualization. They have also been contributing back to the Xen project, so good on them! Instead, they chose to focus on the special version of QEMU included with Xen to bring it up to speed for their product. So they made sure it could do live migration! They also worked around the disk and net I/O issues by creating custom drivers and management software (The aforementioned VSTools) to be installed in the guest after you have the OS running. This limits your choice of OS to run in HVM domains unless you’re willing to build your own VSTools from their recently opened source. They currently support Windows guests up to Windows 2003 Server, and many of the most common “big name” Linux distros. Previously, VirtualIron was using a different proprietary virtualization technology which gave them a chance to develop their management tools into the robust system they are today. They moved over to Xen later but kept their management methodology which works quite nicely with Xen’s best features.
For our solution, we got two big servers for physically hosting our VMs. HP servers with 32 gigs of RAM each, and two dual-core Xeon 64-bit CPUs each. They also have fiber channel interfaces that connect to an HP SAN back-end where we store our HVM domain images. Original I had assumed that I would install VirtualIron on each of these boxes over top of a normal Linux distro, just as I did with Xen kernel installations or any other typical virtualization technology. I did just that and was lost for a bit since this was the completely wrong approach. All it seemed to do was install a DHCP server, a TFTP server, and a Java App server (Jetty if you’re curious) and no Xen kernel. Where was the Xen kernel for the systems to boot into? Digging into their online documentation provided me with a diagram of the architecture of VirtualIron which clarified things considerably. The Java based management interface for VirtualIron contains a “walk through” set up document in a pane on the right hand side of the interface. THAT is where I finally learned and understood the actual architecture and layout. My assumptions were originally completely incorrect. I should have read the manual first!
To use VirtualIron Enterprise (we didn’t go with their Single Server product which DOES work like VMWare and others) you need at least one “management server” and one “managed node”. The management server can be one of a few supported Linux distros, or Windows. The fact that the manager could be a Windows box really confused me at first, because I couldn’t understand how they would get a Xen kernel installed under an already existing Windows installation. (Yes Virtual Iron’s manager can easily be installed on a Windows server to manage the raw hardware nodes.) Again, I was still doubtful about their approach and so was wrong in that line of thinking. But, once I understood the architecture, I was both in awe and very eager to see this thing work. So I proceeded…
The VirtualIron Way:
In my case, I have two managed nodes (those monster servers with 32 gigs each) and one manager (a Xeon dual CPU 32-bit system with 2 gigs of RAM and dual NICs). The manager is running CentOS 4.5, which is supported by VirtualIron as a host for the Enterprise manager. Once I had that installed and had the management network up (you need a separate LAN dedicated to the manager and each node that you can consider “out of band”), I set one of my managed nodes to do a PXE network boot off of the manager. That’s correct, you DON’T need to install a single thing on the managed nodes, they boot via the network from Xen images stored on the manager. It’s all diskless booting via the NICs. The TFTP server and the DHCP server give this box an IP address, and point it to a pre-configured Xen microkernel boot image. Their boot image is a Xen hypervisor with a very stripped down Suse Linux Enterprise 10 (SLES10) on it. So stripped down that the managed nodes can run headless as there is ZERO interaction on those boxes other than the power button.
Once the managed node loads it’s boot from the network, it shows up in the Java management interface and you’re ready to create VMs and assign them RAM, CPU, network and storage. (NOTE: You need to check their hardware compatibility list to see if your intended server hardware is supported) In our case, the SLES10 image has drivers for our Emulex LightPulse fiberchannel HBAs, so LUNs presented by the SAN are fully accessible from within the VirtualIron manager (storage connects to the virtualization nodes, not the manager). Once VirtualIron was up, I was off and running installing RHEL 4.5 for my Zimbra installation and the special drivers and software to improve disk and network performance as well as enable live migration for HVM domains. Performance of the virtual machine was definitely very impressive. Also, keep in mind that the managed nodes that host your HVM domains don’t need to have anything installed on them in any way at all. No OS, no kernel, no boot loader, absolutely nothing. This makes the nodes essentially hot swapable as long as you keep your VM utilization across both low enough that one node can host all VMs. Beyond that, not only do they run headless, but you don’t need ANY storage in them at all if you don’t want it and have a SAN or other remote storage like iSCSI. All VM configuration resides on the managing server. So that’s the system you want backed up reliably. But there is literally nothing on the virtualization nodes at all.
Giving it a Go:
After I got it all up and running and had a Zimbra TEST instance running with VSTools installed, it was time to try a live migration. I opened up the groupware web based client and started some work in it to mimic a standard user. Then I opened the VirtualIron manager and located the Zimbra TEST instance in the Virtual Machines tree. I clicked on it and dragged it to the other physical host to initiate the migration. Then I went back and started working in my Zimbra session while I watched the migration proceed. At this point, the manager tells Xen on the source node to dump the contents of the memory for the domain that will be moved. That memory state is copied to the destination node (the other physical host) and then a final sync is done before the copy is brought up to a live state on the destination and original domain on the source is extinguished.
As this was happening, my Zimbra client continued to function completely normally. An end user wouldn’t notice a thing. For extra points, I actually shut down the physical host that originally contained the migrated domain. Absolutely no effect. It was like the HVM domain was never shutdown or moved. I’ve since made use of the live migration feature for a variety of situtions where I didn’t want down time for services but needed to make a change to hardware. After almost a year of using VirtualIron, I’m very satisfied with it from a performance perspective. VirtualIron is quite powerful and relatively inexpensive compared to other high end solutions. It can bring the power of Xen virtualization to anyone who wants it, even if they’ve never touched Linux at all. I find that to be quite an amazing thing considering Xen’s complexity. So if you have a need to run an unmodified OS (Windows or a supported Linux distribution) on Xen, I would highly recommend the VirtualIron product. However, if you are like I am at home, and run Linux nearly exclusively on the server side, consider the open source version of Xen itself using paravirtualization. Paravirtualization does not have the disk and network performance issues that HVM does.
Filed under: Computer Technology, Main | 1 Comment
Tags: virtualiron, virtualization, xen
Recent Entries
- Computer Graphics and Art: I’ve Been Waiting 25 Years…
- I Might be Moving back to VMWare
- GUIs: Tiled Window Managers
- Enso: Humanize Your Computer
- Useful: Linux Network Block Device Support
- How to Get “From Here to There”: Series Introduction
- Stupid SSH Tricks: Yet Another X11 Protocol Tunneling Tutorial
- Stupid SSH Tricks: Some Essentials
- Stupid SSH Tricks: First Installment
- Virtualization: Commercial Xen (Part 2)
- Virtualization: Xen (Part 1)
Categories
- Architecture (1)
- art (1)
- Books (1)
- Computer Basics (4)
- Computer Technology (12)
- Main (12)
- Reviews (2)
