HOWTO: Resize kvm disk image

I Recently needed a way of re-sizing a kvm disk image. I found quite a few guides and tutorials on this. But i found that several steps were unnecessary and time consuming.


The following is my own recipe on re-sizing a kvm disk image. (some of which is plain copy & paste)


Make backup of kvm disk image
sudo mv disk-image.img /backup-path/disk-image.img.backup


Create a new blank qemu image file
sudo qemu-img create -f raw addon.raw 30G


Append addon.raw to the end of disk-image.img and store as the original disk-image.img file
sudo -s
cat /backup/disk-image.img.backup addon.raw >> disk-image.img


Mount the new disk-image to loop device.
losetup -f (we will assume it returns /dev/loop0)
losetup /dev/loop0 disk-image.img


Then use your favorite partitioning tool (fdisk/parted) to recreate your partitions. take care to ensure the partition numbers and partiton types stay the same, also remember bootable flag.
parted /dev/loop0


Add partition mappings
kpartx -av /dev/loop0


Check filesystem and resize partition
e2fsck -f /dev/mapper/loop0p1 (assuming this is the partition we are extending)
resize2fs /dev/mapper/loop0p1 (assuming this is the partition we are extending)


Recreate filesystem on swap partition
mkswap /dev/mapper/loop0p2 (assuming this is your swap partition)


Remove kpartx mappings
kpartx -dv /dev/loop0


Detach loopback device
losetup -d /dev/loop0

mymediadb.org a free service for managing your media library.

Exactly what is MyMediaDB?

MyMediaDB (mmDB) is a free service for managing your physical and digital media. Create your own media collection of movies, series, books or music and share it with your friends. Its a great tool to keep track of what books you have read. Which tv series you have seen. What movies you have watched and what music you own.

I am the co developer of mmDB and a few days ago we released a preview version of the site. its still riddled with bugs  as its nothing more than a prototype of the final version. The preview version for now, only contains functionality involving movies. You can check the bugtracker for the development progress within tv series, books and music. Our first priority is to get movie and core functionality up and running at an acceptable quality.

If you would like to help us in our effort in creating a online media library solution, please join us in our forum and/or help us submit bug reports and feature requests

Webdesigner feedback tools

I just came across a great article/blogentry on various feedback services geared towards designers. Some of theese tools is a must have for any serious webdesigner!

Take a look yourself.
http://sixrevisions.com/tools/10-excellent-feedback-tools-for-web-designers/

Simple bash script to recursively delete files and folders based on date and freespace

This simple script will automatically free diskspace if the diskspace in the targeted area (specified by $FREEDIR) exceeds the specified limit ($LIMIT). It  will then proceed with recursively removing the oldest file/folder until the used diskpace no longer exceeds the limit.

#!/bin/bash

#CONFIG####################################################

#the dir u wish to watch
FREEDIR="/home/rtorrent/download"

#the limit in percent
LIMIT=90

###########################################################

while [ true ]; do
  USAGE=`df $FREEDIR | tail -1 | awk {'print $4'}`
  USAGE=${USAGE:0:${#USAGE}-1}

  if [ $USAGE -gt $LIMIT ]; then
    TODELETE=$FREEDIR/`ls -tA $FREEDIR | tail -1`
    echo "Removing: $TODELETE"
    rm -rf "$TODELETE"
  else
    break
  fi
done

exit 0

rTorrent Queuing system.

I've created a python script (my very first python project) for handling some simple queuing functinality for rTorrent. When rTorrent is used in conjunction with a rss aggregator you could risk getting too many active torrents which creates overhead in the form of memory, file locks and cpu. The script limits the concurrent active torrents and downloading torrents based on a limit defined in the script config and the creation date of the torrent so that the oldest torrents are removed if any of the limits kicks in.

#!/usr/bin/env python
'''
Created on 27. feb. 2010
@author: Kim Eik
'''

#################################################
#    CONFIG
#################################################

CONNECT_URI = "scgi://localhost:5000"
MAX_LEECHING = 2
MAX_SEEDING = 15

#################################################
#    IMPORT
#################################################

import datetime
from xmlrpc2scgi import do_scgi_xmlrpc_request_py

#################################################
# CLASSES
#################################################

class Torrent:
    def __init__(self, hash):
        self._hash = hash
    def getHash(self):
        return self._hash
    def isStarted(self):
        return self._started;
    def setStarted(self,started):
        self._started = started
    def setCreationDate(self,date):
        self._creationDate = date
    def getCreationDate(self):
        return self._creationDate
    def setCompleted(self, completed):
        self._completed = completed
    def isCompleted(self):
        return self._completed
    def __repr__(self):
        return ("<Torrent %s>" % (self.getHash()))
    def __str__(self):
        return self._hash
        

################################################
#    LOGIC
################################################

torrents = []
rawTorrents = do_scgi_xmlrpc_request_py(CONNECT_URI, "d.multicall", ("","d.get_hash=", "d.get_state=", "d.get_creation_date=","d.get_complete="))
for x,rawTorrent in enumerate(rawTorrents):
    torrent = Torrent(rawTorrent[0])
    torrent.setStarted(rawTorrent[1])
    torrent.setCreationDate(datetime.datetime.fromtimestamp(rawTorrent[2]))
    torrent.setCompleted(rawTorrent[3])
    torrents.append(torrent)
    
downloading = []
started = []
queued = []

for x,torrent in enumerate(torrents):
    if(torrent.isStarted()):
        if(not torrent.isCompleted()):
            downloading.append(torrent)
        else:
            started.append(torrent)
    else:
        if(not torrent.isCompleted()):
            queued.append(torrent)

downloading.sort(key=lambda obj: obj.getCreationDate())
started.sort(key=lambda obj: obj.getCreationDate())
queued.sort(key=lambda obj: obj.getCreationDate())

while(len(downloading) < MAX_LEECHING and len(queued) > 0):
    torrent = queued.pop(0)
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.open",(torrent.getHash(),''))
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.start",(torrent.getHash(),''))    
    downloading.append(torrent)
    print 'starting torrent: ' + torrent.getHash()
    
while(len(downloading) > MAX_LEECHING):
    torrent = downloading.pop(0)
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.stop",(torrent.getHash(),''))
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.close",(torrent.getHash(),''))   
    print 'queueing torrent: ' + torrent.getHash()
    
while(len(started) > MAX_SEEDING):
    torrent = started.pop(0)
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.stop",(torrent.getHash(),''))
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.close",(torrent.getHash(),''))
    print 'stopping torrent: ' + torrent.getHash()
            
        

This script depends on another python script created by Glenn Washburn. xmlrpc2scgi.py which can be fetched from http://libtorrent.rakshasa.no/wiki/UtilsXmlrpc2scgi

You execute this script through a cron job, or through the rtorrent event handling system. Whatever suits your use the best. Good luck!

Using the XBMC eventserver (an alternative).

A while ago i wrote about how you could send notifications to the xbmc event server through a simple C++ program (Using XBMC event server with rtorrent ). However, today i came across an easier way to do the same thing.

Simply put, all you have to do to get the same functionality as described in my previous blogpost is:

sudo apt-get install xbmc-eventclients-xbmc-send
xbmc-send -a "Notification(Testing,XBMC Command Success.)"

Please check out this blogpost for more information.
http://boshdirect.com/blogs/tech/set-xbmc-to-auto-update-library.html

Building the ultimate home IT system. (Part 2)

Virtualization

When it comes to virtualization, you basically have a set of options on how you want virtualize your computers. You can choose to run your virtualized machines through libvirt, which is a a common interface to handle QEMU, KVM and XEN based virtualization methods through the command virsh. In addition when using libvirt you can install virt-manager which gives you a graphical user interface to manage and install virtual machines.

QEMU based virtualization is pure software emulation, where each guest machine has software based virtual devices and kernel. QEMU generates a more overhead than both KVM and XEN.

KVM utilizes virtualization technology present in the central processing unit. So you will need a newer cpu  in order to fully utilize KVM. KVM also uses virtual devices and kernel, but does this in a much more effective way.

XEN on the other hand, don't virtualize the kernel or devices. Instead it runs directly on the host kernel. More on the architecture of XEN can be found here: Xen Architecture_Q1 2008.pdf

XEN and KVM is by far the most effective ways to virtualize computers. The main differences between them is that XEN works on all machines and hardware, while KVM needs virtualization technology present in the cpu. Also afaik, XEN is the only virtualization technology that supports pci/vga passthrough. Enabling you to run for example windows as a virtualized guest with full 3d acceleration as if windows was installed directly on the hardware. (videos and material on this here)

XEN is the technology used in Amazon's EC2 Cloud, but KVM seems to be the mainstream choice of several Linux distro's, like Ubuntu and RedHat.

I have personally tried all of these technologies, and my personal experience was that QEMU was too slow, XEN was a bit flakey for me, getting segmentation faults in several applications running after a period of time. But even though i experienced this problem, doesn't mean that you will. KVM on the other hands seems to perform well.

netsrv

In my setup im running ubuntu 9.10 server with libvirt and kvm, the recommended setup of ubuntu. I have installed 5 virtual machines all with the ubuntu 9.10 server installation. The 5 virtual machines are as explained in part 1, netldap, netmisc, netvpn, netfw and netfiler. In this writing moment, netvpn and netfw are just clean installs and not put too use yet, and as for netfw, i will probably have to reinstall with another distro as smoothwall, and ipcop don't come as an deb package. In addition all virtual machines is running in bridged networking mode, giving each machines their own "external" ip address. For the more technical stuff on how i implemented virtualization, i suggest you take a look at https://help.ubuntu.com/9.10/serverguide/C/libvirt.html as there is no reason why i should repeat what is written the the ubuntu server documentation in this blogentry.

Reference:
https://help.ubuntu.com/9.10/serverguide/C/libvirt.html
http://www.xen.org/
http://www.linux-kvm.org/page/Main_Page
http://www.qemu.org/
http://wiki.xensource.com/xenwiki/XenArchitecture?action=AttachFile&do=get&target=Xen+Architecture_Q1+2008.pdf
http://teo-en-ming-aka-zhang-enming.blogspot.com/