Building the ultimate home IT system. (Part 2)

Virtualization

When it comes to virtualization, you basically have a set of options on how you want virtualize your computers. You can choose to run your virtualized machines through libvirt, which is a a common interface to handle QEMU, KVM and XEN based virtualization methods through the command virsh. In addition when using libvirt you can install virt-manager which gives you a graphical user interface to manage and install virtual machines.

QEMU based virtualization is pure software emulation, where each guest machine has software based virtual devices and kernel. QEMU generates a more overhead than both KVM and XEN.

KVM utilizes virtualization technology present in the central processing unit. So you will need a newer cpu  in order to fully utilize KVM. KVM also uses virtual devices and kernel, but does this in a much more effective way.

XEN on the other hand, don't virtualize the kernel or devices. Instead it runs directly on the host kernel. More on the architecture of XEN can be found here: Xen Architecture_Q1 2008.pdf

XEN and KVM is by far the most effective ways to virtualize computers. The main differences between them is that XEN works on all machines and hardware, while KVM needs virtualization technology present in the cpu. Also afaik, XEN is the only virtualization technology that supports pci/vga passthrough. Enabling you to run for example windows as a virtualized guest with full 3d acceleration as if windows was installed directly on the hardware. (videos and material on this here)

XEN is the technology used in Amazon's EC2 Cloud, but KVM seems to be the mainstream choice of several Linux distro's, like Ubuntu and RedHat.

I have personally tried all of these technologies, and my personal experience was that QEMU was too slow, XEN was a bit flakey for me, getting segmentation faults in several applications running after a period of time. But even though i experienced this problem, doesn't mean that you will. KVM on the other hands seems to perform well.

netsrv

In my setup im running ubuntu 9.10 server with libvirt and kvm, the recommended setup of ubuntu. I have installed 5 virtual machines all with the ubuntu 9.10 server installation. The 5 virtual machines are as explained in part 1, netldap, netmisc, netvpn, netfw and netfiler. In this writing moment, netvpn and netfw are just clean installs and not put too use yet, and as for netfw, i will probably have to reinstall with another distro as smoothwall, and ipcop don't come as an deb package. In addition all virtual machines is running in bridged networking mode, giving each machines their own "external" ip address. For the more technical stuff on how i implemented virtualization, i suggest you take a look at https://help.ubuntu.com/9.10/serverguide/C/libvirt.html as there is no reason why i should repeat what is written the the ubuntu server documentation in this blogentry.

Reference:
https://help.ubuntu.com/9.10/serverguide/C/libvirt.html
http://www.xen.org/
http://www.linux-kvm.org/page/Main_Page
http://www.qemu.org/
http://wiki.xensource.com/xenwiki/XenArchitecture?action=AttachFile&do=get&target=Xen+Architecture_Q1+2008.pdf
http://teo-en-ming-aka-zhang-enming.blogspot.com/

Building the ultimate home IT system. (Part 1)

This is the first part of a series of blog entries describing my journey with setting up the perfect home IT system, consisting of several physical boxes including desktop machines, a set-top box and a server with virtualization. Through this series of blogentries i will describe several alternative technologies that that you can use too incorporate:

• Virtualization (libvirt/KVM)
• File server (Samba/FTP/NFS)
• Software RAID (RAID1, RAID5)
• Logical Volume Manager (LVM)
• Central authentication (LDAP)
• Media center (xbmc/boxee)
• Firewall w/qos (smoothwall/ipcop)

I will try to give a detailed description of the system as a whole, and my experience of how it works as a unit. But I will also go into the implementation of the different individual services, but not in too great detail.

Planning your system.

As depicted in the above image, this was my inital plan of my home system. 

netsrv
netsrv is the physical machine who is running all the virtual computers (netvpn, netmisc, netfiler, netldap and netfw). The reason why i have divided all the system's services in this way is mainly due to flexibility. Where if one of the virtual computers breaks, I only have to reinstall that single virtual pc and it's services. This generates some overhead on processing power of the server, but all within acceptable range.

netldap
You guessed it, this machine handles the LDAP server in addition to a webfrontend for managing ldap records. ldap enabled client machines like Desktop Computers, netmisc, netfiler and netxbmc will lookup this machine to retrieve user and group information to decide access to files and directories as well as system access when logging in (only available on the Desktop Computers).

netfiler
This machine takes care of all shared storage, and can be accessed through NFS, Samba and FTP. NFS uses ldap information natively, and samba and ftp can be configured to use ldap.

netmisc
Contains my miscellaneous services. Mainly rTorrent for downloads and flexget for rss aggregation. Configured in such a way that all users in the torrent group, can upload torrents in a rtorrent watch directory, and access downloaded data (all residing on netfiler).

netxbmc 
Is connected to my living room TV and handles all my entertainment media, like music, movies, series and pictures. This machine has the storage and backup area on netfiler mounted locally through the NFS protocol. However, xbmc does also support the smb/cifs and ftp protocol. It also provides ldap replication, in case the netldap server should break or be unavailable.

netvpn
Using OpenVPN, im trying to configure an easy way to access all the services on the local network from outside my firewall.

netfw
With an installation of smoothwall or ipcop im hoping that i can achieve a secure local network and QoS (Quality of Service). QoS gives you the power to control network communication in regards of what user applications should be a high and low priority. So for example, say your playing an online game, but you are also downloading a couple of torrents, to make sure the game runs as smoothly as possible. You can tell the firewall to prioritize your network traffic higher than the network traffic of the download. 

Desktop Computers 
A set of desktop computers who authenticate users through ldap either directly (linux users) or indirectly with samba (windows users). The desktop computers should have access through NFS, Samba or FTP to a backup area and a storage area. The backup area consists of two partitions on two separate disks mirrored with software raid and added under LVM for flexibility. The storage area is a fault-tolerant software RAID5 added under LVM.  Access to these shared resources are defined through ldap groups, so if user1 is in the group backup, user1 should be able to write to the backup area.


Getting down and dirty.
This was all I had time for today. But ill be back soon with the dirty details on how I implemented these different open systems to create the ultimate home IT system. As you can see it's not something one can do within a hour or two.