Useful: Linux Network Block Device Support

13Jan09

Difficulty Level: Moderate

What is a Network Block Device and Why Would I Want One?

Let me start this entry out by explaining just what a block device is, in case you’re a newer Linux user and you’re not sure. I didn’t know what one was at one point and a quick explanation would have been helpful. In short, block devices are things like hard drives, flash drives, floppy disks, CD-ROMs and even DVDs. On a more complex level, they are devices which get their input/output in the form of data blocks of a certain size in bits or bytes. For the sake of this discussion, we’ll just be thinking of the devices I listed above.

The Linux kernel among it’s many modules (which can be thought of as drivers) has a particular module called ‘nbd’ which stands for Network Block Device. What this means is that you can take almost any block device and present it to the network. This differs from standard Windows file sharing, or Unix NFS in that you’re not presenting a set of files to the network, but the raw device itself. (iSCSI is another way of accomplishing the same thing with a greater degree of reliability and is accessible to Linux through the Linux iSCSI Target and Linux iSCSI Initiator projects.  NOTE: Think of the target as the server, and the initiator as the client.) Although there are other ways of doing this, network block device support is still nothing to ignore. It may not be robust, but it’s still highly useful for non-mission critical tasks where the expense of a SAN is unwarranted.

I originally found out about nbd when I was first starting to work with Xen virtualization. Their project documentation suggested that a convenient way of being able to store virtual machine images on a server was to use nbd support. When I understood that this was the way to a “poor man’s SAN” since Linux software RAID and LVM volumes could be exported with nbd, I then wondered, what else could I export? I decided to experiment and find out. While it’s not perfect because of some I/O control limitations, it’s still quite handy and simple to implement.

Making NBD Work for You:

The first step towards preparing your system for NBD is determining if you need to recompile your kernel.  Fortunately, most of the popular Linux distributions tend to build most of the optional support as kernel modules by default.  You can think of modules as “drivers” for different functionality and hardware support.  In most cases, you should have access to NBD support.  The simplest way to find out is to get to a shell prompt and type: ‘modprobe nbd’.  If you get no error and simply return to the prompt, then type: ‘lsmod | grep -v grep | grep nbd’.  If you see a line indicating that the nbd module is loaded, your kernel has NBD support built in.

However, if you get an error with the ‘modprobe’ command about the module not existing, or you don’t get anything in return for the ‘lsmod’ command, then it’s likely you’ll need to recompile your kernel.  Recompiling the kernel is beyond the scope of this blog entry, but I will link to some resources to get you started in the right direction at the end of this entry.

Three Components to NBD:

There are three components that make up the entirety of NBD for Linux.  The first is the kernel module which allows the kernel to provide an interface to the device for export to/from the network.  The second component is the ‘nbd-server’ application which handles the exporting of the device over TCP/IP.  And finally, the third part is the ‘nbd-client’ application which imports the device on another machine and presents it as /dev/nbX where ‘X’ is a number.  Depending on the distribution you use, you may be able to find a specific package to install the applications.  If not, there is always the source code from the main NBD project site.

Once you have the kernel module loaded and the applications built here is all you need to do to test it and see if it will work for what you need.  This is an example of exporting a raw unpartitioned IDE hard drive over TCP/IP and then importing it on a remote system:

1. On the system containing the hard drive, run the nbd-server command as follows  (Syntax: nbd-server -r <tcp port> /dev/xxx):

nbd-server -r 2000 /dev/hda

2. On the remote system where you wish to import the devicem run the nbd-client command as follows (Syntax: /nbd-client <ip address of system running nbd-server> <matching tcp port number> /dev/nbd0)

nbd-cliet 192.168.1.1 2000 /dev/nbd0

You should then be able to treat /dev/nbd0 on the system where you imported the device as if it were local.  Use ‘fdisk’ to partition the device, format it for use as a file system, or even a paging file.  I’ve used this successfully with remote flash drives, raw hard drives, partitions on hard drives, LVM logical volumes, and even DVD drives for playing movies on devices that don’t have DVD drives.

A few extra notes:

1. Depending on the kernel version, the NBD device nodes might be /dev/nbdX or it might be /dev/nbX where ‘X’ is always a number.

2. There is a one-to-one relationship between the exported device and your chosen TCP port.  That is to say, that if you use port 2000 for /dev/hda and want to export /dev/hdb simultaneously, you’ll need to increment to the next free port.

3. Before randomly chosing ports, it’s a good idea to take a look at the commonly used ports listed in /etc/services as well as run a ‘netstat -an’ to see what ports your particular systems are using.

4. The performance of the DVD export over an 802.11bg link is quite good after an initial buffer period for something like Xine.

5. It’s quite possible to use an imported NBD device as part of a mirror set if you want a pseudo instant copy on a separate machine, but it’s not highly recommended.

Recompiling the Linux kernel (which seems to be a dying art):

http://www.digitalhermit.com/linux/Kernel-Build-HOWTO.htm

Advertisements


No Responses Yet to “Useful: Linux Network Block Device Support”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: