mymediadb.org a free service for managing your media library.

Exactly what is MyMediaDB?

MyMediaDB (mmDB) is a free service for managing your physical and digital media. Create your own media collection of movies, series, books or music and share it with your friends. Its a great tool to keep track of what books you have read. Which tv series you have seen. What movies you have watched and what music you own.

I am the co developer of mmDB and a few days ago we released a preview version of the site. its still riddled with bugs  as its nothing more than a prototype of the final version. The preview version for now, only contains functionality involving movies. You can check the bugtracker for the development progress within tv series, books and music. Our first priority is to get movie and core functionality up and running at an acceptable quality.

If you would like to help us in our effort in creating a online media library solution, please join us in our forum and/or help us submit bug reports and feature requests

Webdesigner feedback tools

I just came across a great article/blogentry on various feedback services geared towards designers. Some of theese tools is a must have for any serious webdesigner!

Take a look yourself.
http://sixrevisions.com/tools/10-excellent-feedback-tools-for-web-designers/

Simple bash script to recursively delete files and folders based on date and freespace

This simple script will automatically free diskspace if the diskspace in the targeted area (specified by $FREEDIR) exceeds the specified limit ($LIMIT). It  will then proceed with recursively removing the oldest file/folder until the used diskpace no longer exceeds the limit.

#!/bin/bash

#CONFIG####################################################

#the dir u wish to watch
FREEDIR="/home/rtorrent/download"

#the limit in percent
LIMIT=90

###########################################################

while [ true ]; do
  USAGE=`df $FREEDIR | tail -1 | awk {'print $4'}`
  USAGE=${USAGE:0:${#USAGE}-1}

  if [ $USAGE -gt $LIMIT ]; then
    TODELETE=$FREEDIR/`ls -tA $FREEDIR | tail -1`
    echo "Removing: $TODELETE"
    rm -rf "$TODELETE"
  else
    break
  fi
done

exit 0

rTorrent Queuing system.

I've created a python script (my very first python project) for handling some simple queuing functinality for rTorrent. When rTorrent is used in conjunction with a rss aggregator you could risk getting too many active torrents which creates overhead in the form of memory, file locks and cpu. The script limits the concurrent active torrents and downloading torrents based on a limit defined in the script config and the creation date of the torrent so that the oldest torrents are removed if any of the limits kicks in.

#!/usr/bin/env python
'''
Created on 27. feb. 2010
@author: Kim Eik
'''

#################################################
#    CONFIG
#################################################

CONNECT_URI = "scgi://localhost:5000"
MAX_LEECHING = 2
MAX_SEEDING = 15

#################################################
#    IMPORT
#################################################

import datetime
from xmlrpc2scgi import do_scgi_xmlrpc_request_py

#################################################
# CLASSES
#################################################

class Torrent:
    def __init__(self, hash):
        self._hash = hash
    def getHash(self):
        return self._hash
    def isStarted(self):
        return self._started;
    def setStarted(self,started):
        self._started = started
    def setCreationDate(self,date):
        self._creationDate = date
    def getCreationDate(self):
        return self._creationDate
    def setCompleted(self, completed):
        self._completed = completed
    def isCompleted(self):
        return self._completed
    def __repr__(self):
        return ("<Torrent %s>" % (self.getHash()))
    def __str__(self):
        return self._hash
        

################################################
#    LOGIC
################################################

torrents = []
rawTorrents = do_scgi_xmlrpc_request_py(CONNECT_URI, "d.multicall", ("","d.get_hash=", "d.get_state=", "d.get_creation_date=","d.get_complete="))
for x,rawTorrent in enumerate(rawTorrents):
    torrent = Torrent(rawTorrent[0])
    torrent.setStarted(rawTorrent[1])
    torrent.setCreationDate(datetime.datetime.fromtimestamp(rawTorrent[2]))
    torrent.setCompleted(rawTorrent[3])
    torrents.append(torrent)
    
downloading = []
started = []
queued = []

for x,torrent in enumerate(torrents):
    if(torrent.isStarted()):
        if(not torrent.isCompleted()):
            downloading.append(torrent)
        else:
            started.append(torrent)
    else:
        if(not torrent.isCompleted()):
            queued.append(torrent)

downloading.sort(key=lambda obj: obj.getCreationDate())
started.sort(key=lambda obj: obj.getCreationDate())
queued.sort(key=lambda obj: obj.getCreationDate())

while(len(downloading) < MAX_LEECHING and len(queued) > 0):
    torrent = queued.pop(0)
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.open",(torrent.getHash(),''))
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.start",(torrent.getHash(),''))    
    downloading.append(torrent)
    print 'starting torrent: ' + torrent.getHash()
    
while(len(downloading) > MAX_LEECHING):
    torrent = downloading.pop(0)
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.stop",(torrent.getHash(),''))
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.close",(torrent.getHash(),''))   
    print 'queueing torrent: ' + torrent.getHash()
    
while(len(started) > MAX_SEEDING):
    torrent = started.pop(0)
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.stop",(torrent.getHash(),''))
    do_scgi_xmlrpc_request_py(CONNECT_URI,"d.close",(torrent.getHash(),''))
    print 'stopping torrent: ' + torrent.getHash()
            
        

This script depends on another python script created by Glenn Washburn. xmlrpc2scgi.py which can be fetched from http://libtorrent.rakshasa.no/wiki/UtilsXmlrpc2scgi

You execute this script through a cron job, or through the rtorrent event handling system. Whatever suits your use the best. Good luck!

Using the XBMC eventserver (an alternative).

A while ago i wrote about how you could send notifications to the xbmc event server through a simple C++ program (Using XBMC event server with rtorrent ). However, today i came across an easier way to do the same thing.

Simply put, all you have to do to get the same functionality as described in my previous blogpost is:

sudo apt-get install xbmc-eventclients-xbmc-send
xbmc-send -a "Notification(Testing,XBMC Command Success.)"

Please check out this blogpost for more information.
http://boshdirect.com/blogs/tech/set-xbmc-to-auto-update-library.html

Building the ultimate home IT system. (Part 2)

Virtualization

When it comes to virtualization, you basically have a set of options on how you want virtualize your computers. You can choose to run your virtualized machines through libvirt, which is a a common interface to handle QEMU, KVM and XEN based virtualization methods through the command virsh. In addition when using libvirt you can install virt-manager which gives you a graphical user interface to manage and install virtual machines.

QEMU based virtualization is pure software emulation, where each guest machine has software based virtual devices and kernel. QEMU generates a more overhead than both KVM and XEN.

KVM utilizes virtualization technology present in the central processing unit. So you will need a newer cpu  in order to fully utilize KVM. KVM also uses virtual devices and kernel, but does this in a much more effective way.

XEN on the other hand, don't virtualize the kernel or devices. Instead it runs directly on the host kernel. More on the architecture of XEN can be found here: Xen Architecture_Q1 2008.pdf

XEN and KVM is by far the most effective ways to virtualize computers. The main differences between them is that XEN works on all machines and hardware, while KVM needs virtualization technology present in the cpu. Also afaik, XEN is the only virtualization technology that supports pci/vga passthrough. Enabling you to run for example windows as a virtualized guest with full 3d acceleration as if windows was installed directly on the hardware. (videos and material on this here)

XEN is the technology used in Amazon's EC2 Cloud, but KVM seems to be the mainstream choice of several Linux distro's, like Ubuntu and RedHat.

I have personally tried all of these technologies, and my personal experience was that QEMU was too slow, XEN was a bit flakey for me, getting segmentation faults in several applications running after a period of time. But even though i experienced this problem, doesn't mean that you will. KVM on the other hands seems to perform well.

netsrv

In my setup im running ubuntu 9.10 server with libvirt and kvm, the recommended setup of ubuntu. I have installed 5 virtual machines all with the ubuntu 9.10 server installation. The 5 virtual machines are as explained in part 1, netldap, netmisc, netvpn, netfw and netfiler. In this writing moment, netvpn and netfw are just clean installs and not put too use yet, and as for netfw, i will probably have to reinstall with another distro as smoothwall, and ipcop don't come as an deb package. In addition all virtual machines is running in bridged networking mode, giving each machines their own "external" ip address. For the more technical stuff on how i implemented virtualization, i suggest you take a look at https://help.ubuntu.com/9.10/serverguide/C/libvirt.html as there is no reason why i should repeat what is written the the ubuntu server documentation in this blogentry.

Reference:
https://help.ubuntu.com/9.10/serverguide/C/libvirt.html
http://www.xen.org/
http://www.linux-kvm.org/page/Main_Page
http://www.qemu.org/
http://wiki.xensource.com/xenwiki/XenArchitecture?action=AttachFile&do=get&target=Xen+Architecture_Q1+2008.pdf
http://teo-en-ming-aka-zhang-enming.blogspot.com/

Building the ultimate home IT system. (Part 1)

This is the first part of a series of blog entries describing my journey with setting up the perfect home IT system, consisting of several physical boxes including desktop machines, a set-top box and a server with virtualization. Through this series of blogentries i will describe several alternative technologies that that you can use too incorporate:

• Virtualization (libvirt/KVM)
• File server (Samba/FTP/NFS)
• Software RAID (RAID1, RAID5)
• Logical Volume Manager (LVM)
• Central authentication (LDAP)
• Media center (xbmc/boxee)
• Firewall w/qos (smoothwall/ipcop)

I will try to give a detailed description of the system as a whole, and my experience of how it works as a unit. But I will also go into the implementation of the different individual services, but not in too great detail.

Planning your system.

As depicted in the above image, this was my inital plan of my home system. 

netsrv
netsrv is the physical machine who is running all the virtual computers (netvpn, netmisc, netfiler, netldap and netfw). The reason why i have divided all the system's services in this way is mainly due to flexibility. Where if one of the virtual computers breaks, I only have to reinstall that single virtual pc and it's services. This generates some overhead on processing power of the server, but all within acceptable range.

netldap
You guessed it, this machine handles the LDAP server in addition to a webfrontend for managing ldap records. ldap enabled client machines like Desktop Computers, netmisc, netfiler and netxbmc will lookup this machine to retrieve user and group information to decide access to files and directories as well as system access when logging in (only available on the Desktop Computers).

netfiler
This machine takes care of all shared storage, and can be accessed through NFS, Samba and FTP. NFS uses ldap information natively, and samba and ftp can be configured to use ldap.

netmisc
Contains my miscellaneous services. Mainly rTorrent for downloads and flexget for rss aggregation. Configured in such a way that all users in the torrent group, can upload torrents in a rtorrent watch directory, and access downloaded data (all residing on netfiler).

netxbmc 
Is connected to my living room TV and handles all my entertainment media, like music, movies, series and pictures. This machine has the storage and backup area on netfiler mounted locally through the NFS protocol. However, xbmc does also support the smb/cifs and ftp protocol. It also provides ldap replication, in case the netldap server should break or be unavailable.

netvpn
Using OpenVPN, im trying to configure an easy way to access all the services on the local network from outside my firewall.

netfw
With an installation of smoothwall or ipcop im hoping that i can achieve a secure local network and QoS (Quality of Service). QoS gives you the power to control network communication in regards of what user applications should be a high and low priority. So for example, say your playing an online game, but you are also downloading a couple of torrents, to make sure the game runs as smoothly as possible. You can tell the firewall to prioritize your network traffic higher than the network traffic of the download. 

Desktop Computers 
A set of desktop computers who authenticate users through ldap either directly (linux users) or indirectly with samba (windows users). The desktop computers should have access through NFS, Samba or FTP to a backup area and a storage area. The backup area consists of two partitions on two separate disks mirrored with software raid and added under LVM for flexibility. The storage area is a fault-tolerant software RAID5 added under LVM.  Access to these shared resources are defined through ldap groups, so if user1 is in the group backup, user1 should be able to write to the backup area.


Getting down and dirty.
This was all I had time for today. But ill be back soon with the dirty details on how I implemented these different open systems to create the ultimate home IT system. As you can see it's not something one can do within a hour or two.