Christoph's last Weblog entries

Entries tagged "kfreebsd".

Backup strategy
27th June 2014

I've been working on my backup strategy for the notebook recently. The idea is to have full backups every now month and then incremental backups in between as fine-grained as possible. As it's a mobile device there's no point in time where it is guaranteed to be up, connected and within reach of the backup server.

As I'm running Debian GNU/kFreeBSD on it, using ZFS and specifically zfs send comes quite naturally. I'm now generating a new file system snapshot every day (if the notebook happens to be online during that day) using cron.

@daily zfs snapshot base/root@`date -I`
@daily zfs snapshot base/home@`date -I`
@reboot zfs snapshot base/root@`date -I`
@reboot zfs snapshot base/home@`date -I`

When connected to the home network I'm synchronizing off all incrementals that are not yet on the backup server. This is using zfs send together with gpg to encrypt the data and then put it off to some sftp storage. For the first snapshot every month a full backup is created. As there doesn't seem to be a way to merge zfs send streams without importing everything in a zfs pool I create additional incremental streams to the first snapshot of last month so I'm able to delete older full backups and daily snapshots and still keep coarse-gained backups for a longer period of time.

# -*- coding: utf-8 -*-

# Config
SFTP_DIR  = '/srv/backup/mitoraj'
SFTP_USER = 'root'
ZPOOL     = 'base'
GPGUSER   = '9FED5C6CE206B70A585770CA965522B9D49AE731'

import subprocess
import os.path
import sys
import paramiko

term = {
    'green':  "\033[0;32m",
    'red':    "\033[0;31m",
    'yellow': "\033[0;33m",
    'purple': "\033[0;35m",
    'none':   "\033[0m",

sftp = None

def print_colored(data, color):

def postprocess_datasets(datasets):
    devices = set([entry.split('@')[0] for entry in datasets])

    result = dict()
    for device in devices:
        result[device] = sorted([ entry.split('@')[1] for entry in datasets
                                    if entry.startswith(device) ])

    return result

def sftp_connect():
    global sftp

    host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
    hostkeytype = host_keys[SFTP_HOST].keys()[0]
    hostkey = host_keys[SFTP_HOST][hostkeytype]

    agent = paramiko.Agent()
    transport = paramiko.Transport((SFTP_HOST, 22))

    for key in agent.get_keys():
            transport.auth_publickey(SFTP_USER, key)
        except paramiko.SSHException:

    sftp = paramiko.SFTPClient.from_transport(transport)

def sftp_send(dataset, reference=None):
    zfscommand = ['sudo', 'zfs', 'send', '%s/%s' % (ZPOOL, dataset)]
    if reference is not None:
        zfscommand = zfscommand + ['-i', reference]

    zfs = subprocess.Popen(zfscommand, stdout=subprocess.PIPE)

    gpgcommand = [ 'gpg', '--batch', '--compress-algo', 'ZLIB',
                   '--sign', '--encrypt', '--recipient', GPGUSER ]
    gpg = subprocess.Popen(gpgcommand, stdout=subprocess.PIPE,

    if gpg.returncode not in [None, 0]:
        print_colored("Error:\n\n" + gpg.stderr, 'red')

    if reference is None:
        filename = '%s.full.zfs.gpg' % dataset
        filename = '%s.from.%s.zfs.gpg' % (dataset, reference)

    with, 'w') as remotefile:
        while True:
            junk =*1024)
            if len(junk) == 0:

        print_colored(" DONE", 'green')

def syncronize(local_datasets, remote_datasets):
    for device in local_datasets.keys():
        current = ""
        for dataset in local_datasets[device]:
            last = current
            current = dataset

            if device in remote_datasets:
                if dataset in remote_datasets[device]:
                    print_colored("%s@%s -- found on remote server" % (device, dataset), 'yellow')

            if last == '':
                print_colored("Initial syncronization for device %s" % device, 'green')
                sftp_send("%s@%s" % (device, dataset))
                lastmonth = dataset

            if last[:7] == dataset[:7]:
                print_colored("%s@%s -- incremental backup (reference: %s)" %
                              (device, dataset, last), 'green')
                sftp_send("%s@%s" % (device, dataset), last)
                print_colored("%s@%s -- full backup" % (device, dataset), 'green')
                sftp_send("%s@%s" % (device, dataset))
                print_colored("%s@%s -- doing incremental backup" % (device, dataset), 'green')
                sftp_send("%s@%s" % (device, dataset), lastmonth)
                lastmonth = dataset

def get_remote_datasets():
    datasets = sftp.listdir()
    datasets = filter(lambda x: '@' in x, datasets)

    datasets = [ entry.split('.')[0] for entry in datasets ]

    return postprocess_datasets(datasets)

def get_local_datasets():
    datasets = subprocess.check_output(['sudo', 'zfs', 'list', '-t', 'snapshot', '-H', '-o', 'name'])
    datasets = datasets.strip().split('\n')

    datasets = [ entry[5:] for entry in datasets ]

    return postprocess_datasets(datasets)

def main():
    syncronize(get_local_datasets(), get_remote_datasets())

if __name__ == '__main__':

Rumors have it, btrfs has gained similar functionality to zfs send so maybe I'll be able to extend that code and use it on my linux nodes some future day (after migrating to btrfs there for a start).

Tags: foss, gnupg, kfreebsd.
A week of Debian GNU/kFreeBSD
4th September 2011

While other people are squashing RC bugs I was using this week to fix (or investigationg) some more kFreeBSD issues -- mostly looking at failed build logs and trying to fix the problems and after some nice fish for dinner writing things up.

Tags: debian, foss, kfreebsd, programmieren.
29th June 2011

debconf banner I'm coming as well! Really looking forward to meet the people from Debconf9 again. Also people from the Games Team and the buildd and kfreebsd folks. And ideally there will be some more people interested in (Common) Lisp as well, we'll see

Tags: debian, foss, kfreebsd, linux.
Maintaining kFreeBSD buildds since one month
30th May 2011

At April 30, I took over maintenance of of Debian's kFreeBSD autobuilders. Means getting something like 4,5k e-Mails this month (gladly no need to sign all those 4k successful builds any more!), filling nearly 30 RC Bugs (quite a lot of which got fixed just within hours after filling, wow!), investigating some of the more strange build failures and such stuff. In general it turned out to be quite some fun.

Quite interesting which libraries turn out to be rather central to the Archive. I wouldn't have guessed that a uninstallable libmtp would prevent a reasonable fraction of the builds to fail -- including packages like subversion.

Packages builds failing because the disk space is exhausted may be something most of us have already witnessed, especially those here that use one of these small notebook hard drives. Build failures caused by a lack of RAM might certainly be imaginable as well, especially on highly parallel builds. But have you ever seen gcc fail because the virtual address space was exhausted on 32 bit architectures?

Also there's a interesting lot of packages with misspelled build dependencies which sbuild can't find and therefore can't build the package. Maybe having a lintian check for some of these problems would be a good idea?

I'm also regularly seeing build failures that look easy enough to fix -- like some glob in a *.install for some python package matching lib.linux*. I try to fix some of these as I see them but my time is unfortunately limited as well. Someone interested in quick&easy noticed about these kind of issues? I could put URLs to build-logs on or somewhere on IRC.

There are also some really strange failures like llvm, which builds flawlessly on my local kFreeBSD boxes all the time, inside and outside schroot but hangs all the time in the same testcase when building on any of the autobuilders (any hints welcome!) or perl failing on kfreebsd-amd64 selectively but all the time.

Tags: debian, foss, kfreebsd, porting.
Debian GNU/kFreeBSD
29th December 2010

So when I was travveling to my parent's for christmas it looked like I'd have limited computer access. My Netbook is quite OK for reading mail but not really useable for any real hacking. And my trusty Thinkpad (Z61m) was oopsing when X was running so not much useable either. But as some Live CDs placed here were working well I decided that this would be fixed by an reinstall. And as I was reinstalling anyway I decided I could just choose kfreebsd-amd64. Turned out to be a quite entertaining decision with lots of stuff to hack away with


Bad news: there's no wireless support on Debian GNU/kFreeBSD at the moment. This problem is tracked as Bug #601803 so for wireless internet you will need a (plain) Freebsd chroot. Haven't tried this myself yet -- busy figuring other stuff out.


Having a FreeBSD chroot I decided to give SBCL on GNU/kFreeBSD another try after having failed to get it working in a VM some time ago. With quite some help on SBCL's IRC channel I managed to build a patch that enables building (you need to force a :os-provides-dlopen to the feature list additionally).

There's currently no multi-threading working so I hae a project for the rest of the hoidays (well lots of other stuff to do as well ;))


Some more user-related stuff now. As it is this time of the year I wanted to listen to some 27c3 streams so I needed working audio. However there's no OSS device available. Turned out you just need to kldload the right module (here snd_hda) to get sound working.

Volume was rather low although hardware controls of the soundcard where at max. As that's all OSS there's no point looking for alsamixer. Turns out aumix can do that here.

IPv6 aiccu stuff

Installing aiccu, copying the config in and starting did not work out as well. I already tried to do that from within the FreeBSD chroot already (which doesn't work for some reason) until I discovered just loading the if_tun kernel module solves the aiccu on Debian issue quite well. To get a default route up the last step was finding /lib/freebsd/route again -- /sbin/route is a wrapper around that abstracting differences in BSD route but not supporting IPv6.

Tags: debian, foss, kfreebsd, programmieren.
Introducing Debian GNU/kFreeBSD
1st September 2009

Gute Nachrichten vorneweg: Debian GNU/kFreeBSD funktioniert bereits halbwegs und ist mehr oder weniger verwendbar. Zumindest für meine Zwecke. Und trotzdem wird mein Notebook vorerst weiter mit Debian GNU/linux betrieben. Ist einfach die stabilere Variante.

Das heißt jetzt natürlich nicht, kFreeBSD einfach zu ignorieren und daher habe ich eine Installation in QEMU laufen. Im folgenden ersteinmal ein paar Screenshots:

Debian kFreeBSD mit Iceweasel und urxvt auf fluxbox Desktop

Während der X Server weitgehen funktioniert -- neueste Kernel Version vorausgesetzt und mitunter etwas nachhelfen an den HAL fdi Dateien -- ist das mit den Desktops etwas schwieriger. Fluxbox, wie oben zu sehen, funktioniert dabei einwandfre und auch awesome scheint zu funktionieren:

Debian kFreeBSD mit Iceweasel auf awesome Desktop

Während ich persönlich mit Fluxbox bisher immer ganz gut zurecht gekommen bin und mir awesome in der QEMU ziemlich gut gefällt (muss ich wohl nochmal auf dem Notebook direkt ausprobieren), ist meine erste Empfehlung an Linux Neulinge normalerweise LXDE. Und da fangen die Probleme an.

Zuerst schon lässt sich lxde, und sogar lxde-core nicht direkt installieren, nicht erfüllbare Abhängigkeiten. Glücklicherweise lässt sich das für lxde-core relativ einfach beheben: für pcmanfm ist der hal build auf nicht-Linux Architekturen (Hurd, BSD) deaktiviert. Schaltet man ihn an, baut das ganze frühlich das pcmanfm Packet.

lxpanel ist etwas komplizierter, lässt sich aber auch beheben. Das Packet hängt zwar für den Bau von libasound2-dev und libiw-dev ab, die (noch) nicht auf kFreeBSD portiert wurden, allerdings erkennt der configure script das automatisch und baut dann halt ohne. Nur das erkennen der BSD Kernels funktioniert nicht ganz, auf GlibC BSDs definiert GCC nämlich __FreeBSD_kernel__ statt __FreeBSD__. Passt man die entsprechende Zeile im Quellcode an funktioniert zumindest der Build. lxmusic müsste ich mir 'mal ansehen, auch hier gibt es unerfüllte Build-Abhängigkeiten.

Versuch, lxterminal in LXDE auf Debian GNU/kFreeBSD zu öffnen

Auf dem Screenshot ist schon ganz gut eines der Probleme zu sehen: Der Dekorator will irgendwie nicht so recht. Und in Live oder auf einem Video könnte man auch das zweite Phenomen erkenne: Das halbdekorierte Fenster wandert langsam immer weiter den Bildschirm hinunter.

GNOME und/oder KDE konnte ich nicht so einfach testen. Sowohl gnome-core, als auch kde4-minimal wollten sich wegen nicht erfüllter Abhängigkeiten nicht installieren lassen und für GNOME/KDE fehlt mir auch ersteinmal die Motivation etwas tiefer zu graben.

Zu guter letzt bleibt noch zsh. Hier hängt der Build auf den Debian Autobuildern im test Status. Abhilfe schafft ein lokaler Build, für den die Tests auskommentiert sind (leider unterstützt das Packet kein DEB_BUILD_OPTIONS="notest".

Jetzt steht eigentlich die Installation auf echter Hardware an, allerdings braucht der PC, den mir der Lukas freundlicherweise überlassen hat, wohl ersteinmal noch ein BIOS update um mit Linux und/oder BSD Kernels zurechtzukommen.

Für alle, die neugierig geworden sind, gibt es eine Mailing Liste und einen IRC Channel sowie (leider nicht ganz aktuelle Informationen auf Alioth.

Tags: debian, foss, kfreebsd, programmieren.

RSS Feed

Created by Chronicle v4.6