Posts – Page 2

Summer of Code 2011

The Samba team is once again participating in the Summer of Code this year. This year we have 4 students working on various projects related to Samba.

This year I am mentoring Dhananjay Sathe, who is improving the GTK+ frontends for Samba. In particular, he is making it possible to manage shares and users of a remote Samba or Windows machine.

Dhananjay is also blogging about his progress.

comments.

libapache2-mod-bzr

During the last two days I hacked together a Bazaar module for Apache. This module makes it possible to easily enable the Bazaar smart server for Bazaar branches. It also can display a simple placeholder page for Bazaar branches without working tree. It's surprisingly easy to write Apache modules.

The main advantage this has over a mod_wsgi / mod_python / mod_fcgi setup is that it doesn't require any additional Python hacking on the users side or other configuration outside of Apache, and it doesn't require configuration for each single branch in the Apache configuration. In the future I'd also like to support the settings "BazaarFrontend Wikkid" and "BazaarFrontend Loggerhead".

The configuration is currently as simple as:

LoadModule bzr_module /usr/lib/apache2/modules/mod_bzr.so
BazaarSmart on
BazaarFrontend Basic

in your apache2.conf. The BazaarSmart and BazaarFrontend directives can appear in <Directory> or <Location> clauses as well, if you'd like to have different behaviour for different directories.

At the moment this project is a proof of concept, and probably not something you would want to run in production. For example, there is no way to limit the access to a branch to read only. I need to double-check there are no threading issues.

Testing and patches are welcome. The project is hosted here:

Currently Playing: Stream of Passion - Calliopeia

comments.

On the way to Samba 4: Part 2

It's been more than a month since the last status update on my Samba 4 work - much more than the two weeks I promised.

During the holidays I finally managed to release the new alpha of Samba 4, as well as releases of some of our companion libraries (tdb, talloc, tevent and ldb). The release includes a significant amount of bug fixes and a lot of work towards a properly functioning Active Directory DC, too much to list here.

This release I've mainly been involved in improving our Python bindings and our handling of internal and external libraries. We now use symbol versioning for our copy of Heimdal Kerberos as well as some of our other libraries. Hopefully this will fix some of the issues users of the evolution-mapi plugin have been seeing where they end up with both MIT Kerberos and Heimdal Kerberos loaded into the same process (with all the consequences of overlapping symbol names). Samba 4 now also has the ability to work with the system Heimdal rather than using the bundled copy. I have packaged alpha14 for Debian and Ubuntu (fixing most of the open bugs against the Samba 4 package in the BTS), but am currently waiting for the new release of ldb to pass through NEW before I can upload.

The next release is scheduled for the first week of February.

Currently Playing: Stream of Passion - Haunted

comments.

On the way to Samba 4: Part 1

After Samba XP 2008 Andrew and I started keeping a wiki page with our bi-weekly goals and achievements for Samba 4. Because planning in a Free Software project is hard (time availability and priorities change over time, and other volunteers are equally unpredictable) we called this our "Fantasy Page"; it listed things we wanted to work on next ("fantasies"), but reality being what it is we would usually actually end up working on something entirely different. We discussed our progress and new plans in - what I would now call - a bi-weekly standup call.

There were several reasons for doing this. It gave us some sense of direction as well as a sense of accomplishment; a way to look back at the end of the year and realize how much we had actually achieved. Because Samba 4 is such a long term project (it is 7 years old at this point) it is easy to become disillusioned, to look back at a year of commits and to not see the gradual improvement, just the fact that there is no release yet.

We managed to keep this up for two years, much longer than I had anticipated, and eventually started to slip last year.

More recently Kai and Tridge have started to blog weekly about their efforts to make Samba 4.0 a reality and I'm going to join them by trying to blog regularly - every two weeks - about my contributions, even if there were none.

In the next two weeks I plan to work on finally getting alpha 14 of Samba 4 out and on fixing the daily builds of Samba 4 and OpenChange for Ubuntu on Launchpad after we did a massive reorganization of the private libraries in Samba 4.

Current Playing: Zero 7 - Somersault

comments.

Mumble and bluetooth

Mumble is an open source, low-latency, high quality voice chat application that we're using at Canonical, and which Samba has recently also adopted.

After I busted the cable of my cabled headset and inspired by Matt's post about Mumble and Bluetooth I bought a cheap Nokia Bluetooth headset that works with my laptop as well as my phone. By using BlueMan I even found that the headset worked out of the box on Maverick.

A nice feature in Mumble is that it is controllable using D-Bus. Blueman supports running an arbitrary command when the answer button is pressed. Combining these two features, it is possible to automatically toggle mute when the answer button is pressed:

#!/usr/bin/python
import dbus
bus = dbus.SessionBus()
mumble = bus.get_object("net.sourceforge.mumble.mumble", "/")
is_muted = mumble.isSelfMuted()
mumble.setSelfMuted(not is_muted)

To use this script, set its path in the configuration tab for the "Headset" plugin in blueman.

The only remaining problem with this setup is that I can't keep the headset on during my work day, as I don't have a way to put it in standby mode automatically. This means that my battery runs out pretty quickly, even when nothing is happening on Mumble.

Currently Playing: Red Sparowes - Finally As That Blazing Sun Shone

comments.

OpenChange server and SOGo

There's more good news on the OpenChange front. Julien has been working together with Wolfgang and Ludovic from Inverse to leverage the server-side support in OpenChange to provide native Exchange server support in SOGo.

A couple of days ago we announced that there now is an initial version that allows the use of Outlook against a SOGo server through OpenChange.

There is a screencast up on youtube (there is also a .mov version of the screencast).

As far as I know, this is the first time it's possible to actually use Outlook clients with a non-Microsoft Exchange-compatible server, without the need for plugins on the Outlook side. And it's all Free Software. Of course, this is just a preview, and not something we'd recommend everybody to run in production just yet. But it's exciting to finally see this come together.

We already have OpenChange packages in Debian and Ubuntu but I hope I can help get SOGo packaged for both distributions as well.

comments.

Samba 4 and OpenChange daily Ubuntu packages

Daily builds

As of a month ago there are Ubuntu archives with fresh packages of Samba 4 and OpenChange, built on a daily basis day from the latest upstream revision.

This means that it is now possible to run a version of Samba 4 that is less than 24 hours old, without having to know how to extract source code from the version control system that upstream is using, without having to know how to build and install an application from source, but perhaps most importantly: without having to go through the tedious process of manually updating the source code and rebuilding.

OpenChange is tightly coupled to Samba 4, so installing a new version of OpenChange usually involves installing a new version of Samba 4 as well. To make matters more confusing, the two projects use different version control systems (Samba 4 is in Git, while OpenChange is in Subversion) and different build systems (Samba 4 uses waf, OpenChange uses autoconf and make).

I have been involved in Samba 4 and OpenChange as an upstream developer and more recently also as a packager for both Debian and Ubuntu.

As an upstream developer for both these projects it is important for me that users can easily run the development versions. It makes it possible for interested users to confirm the fixes for issues they have reported and to test new features. The more users run the development version, the more confident I can be as a developer that doing a release will not cause any unexpected surprises.

As a packager it is useful to know when there are upstream changes that are going to break my package with the next release.

Recipes

The daily builds work using so-called recipes which describe how to build a Debian source package from a set of Bazaar branches. For example, the Samba 4 recipe looks like this:

# bzr-builder format 0.2 deb-version 4.0.0~alpha14~bzr{revno}~ppa{revno:packaging}+{revno:debian}
lp:samba
merge debian lp:~samba-team/samba/unstable
merge packaging lp:~samba-team/samba/4.0-ppa-maverick

This dictates that a source package should be built by taking the upstream Samba branch and merging the Debian packaging and some recipe-specific tweaking. The last bit on the first line indicates the version string to be used when generating a changelog entry for the daily build.

Every night Launchpad (through bzr-builder) merges these branches and attempts to build the resulting source package, e-mailing me in case of build problems. Generally I fix issues that come up by committing directly to upstream VCS or to the Debian packaging branch. There is no overhead in maintaining the daily build after I've set it up.

For more information on creating source package recipes, see getting started.

Toolchain

The entire toolchain that does the daily package builds for Ubuntu is Free Software, and I have contributed to various bits of that toolchain over the years. It's exciting to see everything come together.

Soyuz

Launchpad consists of multiple pillars - one of those pillars is Soyuz, which I hack on as part of my day job at Canonical. Soyuz is responsible for the archive management and package building. Debian source packages (a combination of upstream source code and packaging metadata) get uploaded by users and then built for various architectures on our buildfarm and published to the Ubuntu archive or to users personal package archives.

Launchpad-code

Another pillar of Launchpad is Launchpad-code, which is responsible for the hosting and management of version control branches. Launchpad users can either host their branches on Launchpad directly or mirror branches (either native Bazaar branches or branches in a foreign format such as Subversion, Git or Mercurial). The mirrorring of native and foreign branches happens using standard Bazaar API's. In the case of Samba and OpenChange we import the branches of the upstream projects (Samba is in Git, OpenChange is in Subversion) and the packaging for both projects is in Bazaar.

Launchad-code calls out to Bazaar to do the actual mirrorring. Over the last few years I have done a lot of work to improve Bazaars support for foreign branches, in particular on supporting Subversion, Git and Mercurial. As the code mirrorring in Launchpad is one of the biggest users of bzr-svn and bzr-git it has helped find some of the more obscure bugs in those plugins over the last few years, to the point where there are only a handful of issues with Git imports and Subversion imports left.

bzr-git and dulwich

bzr-git provides transparent access to Git repositories from within Bazaar and is built on top of Dulwich. Dulwich is a Python library that provides access to the Git file formats and protocols that is completely independent of Bazaar. James Westby originally started it and I adopted it for bzr-git and further extended it. There are now several other projects that use it as well, including hg-git, and rabbitvcs. Apart from James and myself, almost two dozen other people have contributed it so far.

bzr-svn and subvertpy

bzr-svn provides transparant access to Subversion repositories in Bazaar. When I grew frustrated with the existing Subversion Python bindings for various reasons, I decided to create independent Python bindings for Subversion from scratch. These bindings have since been split out into a separate project - subvertpy - and other projects have since also started using them, e.g. hgsubversion and basie.

Using the daily builds

To use the Samba 4 and OpenChange daily builds (Ubuntu Maverick only for now), run:

$ apt-add-repository ppa:samba-team/ppa
$ apt-add-repository ppa:openchange/daily-builds

Currently Playing: Karnivool - Themata

comments.

subunit usage in Samba

Both Samba 3 and Samba 4 are now using the "subunit" protocol inside their testsuite (aka "make test"). subunit is a streaming protocol used to report test results that is aimed at being simple to generate and parse as well as being human readable.

A very simple subunit stream might look like this:

test: samba4.tests.util.strlist.check_list_make
creating list...
list created!
success: samba4.tests.util.strlist.check_list_make
test: samba4.tests.util.strlist.check_list_make_shell
creating list...
xfail: samba4.tests.util.strlist.check_list_make_shell [
returned NT_STATUS_NOT_IMPLEMENTED
]

For those that are familiar with the TAP protocol used by Perl, it is similar to that, although it has a couple of features that TAP does not have. For example, it can report timestamps (useful for determining test duration) and has more flexible progress reporting.

Subunit is particularly useful for projects that use multiple programming languages as it allows a single tool to be used for test visualization or analysis rather than one per language. All that's required per-language is a test runner that can spit out subunit streams.

selftest.pl, the main engine behind Samba's test suite, has been using subunit internally since its creation a couple of years ago. Most other test tools we use can also report subunit, in particular our Python tests, blackbox tests, Perl tests (using tap2subunit) and smbtorture.

"make test" never displays raw subunit results, it always formats them using our format-subunit script. Samba 4's "make test" stores the raw subunit output in st/subunit.

I'm attending SNIA SDC at the moment and a couple of people here have asked me about the tools I use to display and analyse test results. They are:

subunit

The subunit project contains a bunch of convenience tools for working with subunit. Other than libraries for parsing/generating subunit for several languages it contains tools for manipulating and analysing subunit streams, including:

  • subunit-ls: List all tests in a subunit stream, optionally including their run times (I used this for the test duration summary I sent to the Samba mailing list earlier)
  • tap2subunit: convert a TAP stream to a Subunit stream
  • subunit-stats: Print statistics for a subunit stream (how many successful tests, failed tests, skipped tests, etc)
  • subunit-filter: E.g. remove test result or output from a stream
  • subunit-diff: Compare two subunit streams and see what tests have started failing or are no longer failing
  • subunit2pyunit: Format a subunit stream using Python's standard unit test test result formatter

We're including the subunit tree in the Samba git tree at lib/subunit.

tribunal

Tribunal is a GTK+ viewer for subunit streams. It allows for easy browsing of test results. Tribunal is still a bit rough around the edges, although it should already be useful.

Example usage:

$ make test
$ tribunal-subunit st/subunit

testrepository

Test Repository provides a database of test results which fits into developers work flow and keeps track of useful information like what tests are failing, or which failures have the same backtrace.

In particular Test Repository can re-run only the tests that failed in the previous test run:

$ testr init
# Run the full testsuite (1 hour goes by)
$ testr run
# Run those tests from the testsuite that failed in the previous run
# (this would be a lot shorter usually, depending on how many tests were
# failing)
$ testr run --failing

testrepository is also still in its early days, but can potentially be very useful, e.g. when comparing old test runs on the buildfarm.

comments.

Working from home

For about 6 months now I've been working for Canonical on the Soyuz component of Launchpad. Like most other engineers at Canonical I don't work at the office but from a desk at home, as our nearest office is in London, not really a distance that is feasible for a commute. I do work at regular hours during work days and stay in touch with my colleagues using IRC and voice over IP.

I did have some experience working on contracts and study assignments from home previously, but working a fulltime regular job has turned out to be a bigger challenge. It seems easy enough. No travel time, every day is casual Friday, being able to listen to obscure death metal all day without driving coworkers crazy. Awesome, right?

Well, not entirely. I can't say I wasn't warned beforehand (I was) but I still ran head-first into some of the common mistakes.

Solitude

I can work well by myself and I appreciate the occasional solitude, but it does get kinda lonely when you're physically sitting by yourself for 8 hours a day, five days a week.

Fortunately we regularly have sprints at different locations around the world and, apart from appealing to the travel junkie in me, that brings some essential face time with coworkers. Electronic communication mechanisms such as mailing lists, IRC, Skype and, more recently, mumble also help make the rest of the company feel closer, but it's still very different from being able to talk to people at the water cooler (the point of which, btw, still escapes me. What's wrong with proper cold tap water?).

What also seems to help is going into the city and meeting up with others for lunch, or even just to get groceries.

Concentration, work times

One of the nice things about working at home is that you're quite flexible in planning your days; it's possible to interrupt work to run an errand if necessary. The downside of it is that it is also really easy to get distracted, and there's something I do very well: procrastinating. I initially ended up getting distracted quite often and then would end up working into the evening to make up for that lost time. The result being that, while only spending 8 hours doing actual work, it felt like having been at work for 12 hours in the end and having lost all spare time. Or as a friend summarized it accurately: working at home is all about boundaries.

This is at least partially related to the fact that I am a compulsive multi-tasker; I always do several things at once and context-switch every minute or so (prompted by e.g. having to wait for code to compile), including checking email and responding to conversations on IRC and Google Talk. This, among other things, has the effect that I respond quite slowly in IRC/IM conversations; if you've ever chatted with me you've probably noticed it. Multi-tasking has always worked well for me - despite research suggesting otherwise - because software development always involves a lot of waiting (for vcses, compilers, testsuites, ...).

Recently I've tried to eliminate some of the other distractions by signing out off Skype, Empathy (Google Talk, MSN, etc) and Google reader completely and only checking email a couple of times per day.

Feeling productive

What has perhaps surprised me most of all was how essential the satisfaction of getting something done is. After spending about a day staring at Python code it's important for your mood to have accomplished something. This appears to be a vicious circle, as lack of progress kills the fun of work, which kills motivation, which causes a lack of progress.

I am hard core, so during my first months I used my lunch breaks and evenings to hack on other free software projects, triaging bug reports that had come in or reviewing patches. Despite the fact that this is indeed technically a break from Launchpad, it didn't (surprise!) seem to work as well as stepping away from the computer completely. Also, it turns out that spending 14 hours a day programming doesn't make you all that much more productive than working a couple of hours less.

What I've discovered recently is that getting at least one branch done by the end of each day, even if it's just by fixing a trivial bug, helps tremendously in giving me some sense of accomplishment. Julian also wrote a blog post with some useful hints on feeling productive a while ago.

What is your experience working from home? Any good tips?

Currently Playing: Sieges Even - Unbreakable

comments.

Samba Summer of Code

As I have done in previous years, I am again participating in the Google Summer of Code as mentor for the Samba project.

Last year I Andrew and I co-mentored three students with mixed results. In the end we had to drop one of our students but the other two did well. I've only taken on one student this year for various reasons.

The amount of time required to mentor a student varies wildly depending on the student and is hard to predict based on their application. Some students seem to require quite a lot of mentoring while others are self-motivated and self-learning. This has not just been my experience, I've heard similar stories from fellow mentors on other projects.

Last summer Ricardo worked on SWAT for Samba 4 and he is still actively working on the project, even after the Summer of Code has finished. I hope to find the time to package SWAT in time for Debian Squeeze. At the moment SWAT just supports managing shares but Ricardo is working on user management.

In 2009 Calin worked on the GTK+ frontends for Samba, in particular changing them to be Python-based rather than C-based. This year his work is going to be continued by Sergio, hopefully with the some user-ready tools as the end result.

Currently Playing: Gazpacho - 117

comments.

Proof of concept OpenChange server working

Seeing this makes me very happy. It's taken us a couple of years to get to this point but we've finally made it, mostly thanks to the dedication and persistence of Julien and Brad.

comments.

Thoughts on the Nokia N900

Yesterday I started writing up a quick review of my new phone/tablet, a Nokia N900. Unfortunately it has since broken down by what appears to be a hardware issue. It does still work to a certain degree, as the leds flash and I can access it as a USB mass-storage device, but the screen hasn't shown anything since I woke up this morning. The timing could not have been worse, as I'm currently abroad and having a phone (as well as a GPS) would be useful, especially considering the ash cloud situation in Iceland.

Update: 4 weeks after dropping my phone off at the Nokia care center I have received a new N900; they weren't able to tell me what was wrong with the old one.

Anyway, my thoughts about the N900. Please note that I haven't used a lot of other smartphones, so I can't really compare this with Android phones or iPhones.

The good

The integration between the different components of the phone is really well done. Nokia has invested heavily in Telepathy, which is used for all voice and text messaging, and it shows. Skype and regular telephony are very nicely integrated with telepathy, albeit through proprietary daemons. There is one global status for all protocols. Skype and SIP conferencing work well.

The address book is another thing that is very nicely integrated with the rest of the device. It combines IM addresses, phone numbers, email addresses and other information for a contact and there are third party applications that can help keep that information up to date.

Both the Ovi maps app and its data are free, and it's possible to copy the maps onto the device so you don't need to stream them all from the internet when you're abroad. It's pretty quick compared to e.g. a Garmin GPS, but lacks features. It can't load GPX files, can't show points of interest, store tracks, etc.

The music player is neat and automatically indexes all audio and video files that are copied onto the device. The camera and photo manager work well, and are reasonably snappy.

The web browser displays all pages I've accessed so far without rendering issues, including flash.

There are quite a lot of neat third-party free software applications available - eCoach, Hermes, grr, mBarcode, Mauku, tubes, tuner, fahrplan.

The bad

Maemo is Linux-based and it has a lot of similarities to your average Debian-derived distribution. Despite that, it contains a lot of proprietary software. In particular, the Skype and POT plugins for telepathy, the address book and things like Ovi Maps are proprietary. This means it's impossible for me to fix little annoyances (see below) in these packages, but more importantly, it's impossible for others to fix these little annoyances or port these apps to the desktop and other devices (where there's an even larger pool of potential bug fixers).

The GPS doesn't work very well. This might be due to the fact that the hardware is substandard, but I suspect it has got more to do with the fact that if it does not find a signal within 30 seconds it will switch itself off. I assume this is to save energy, but this behaviour can not be switched off anywhere. A workaround is to close and reopen Ovi Maps every 15 seconds or so, but that's pretty annoying. I'm sure somebody with access to the source (I don't!) can fix it.

I'm sure the calendaring app is nice, but it lacks support for synchronization with Google calendar, which makes it unusable for me.

The standard email client does work, but it doesn't scale well. It will lock up at times while trying to index mailboxes. I've tried using claws mail for a while, but its interface is just too cumbersome - in general, but in particular on such a small screen.

The battery lifetime sucks. If I'm lucky I can make it through a single day with a fully charged battery, with only mild usage. The form factor is also still a minor issue for me, but I doubt that bit can be fixed in software.

Overall I'm pretty happy with the N900, although I'm not sure if I would pick it over an Android phone again.

comments.

Nostalgia: 10 Years of Samba Hacking

While searching for something else I happened to come across one of my first posts to the ntdom list in November 2000.

My post is a simple question about a Samba crash that I myself no doubt had introduced. I'm sure I could have found a solution to it by using Google - excuse me, AltaVista - but I still received a friendly reply from Jerry explaining me to use GDB. I'm not too embarrassed, at least I used proper punctuation and already wrote somewhat comprehensible English back then.

It's also strange to realize it's already been almost ten years since I started hacking on the Samba project.

comments.

Linux.Conf.Au 2010 - Day 3 - Wednesday

I went to Jonathan Corbet's yearly update of the status of the Linux kernel. He talked about the various big changes that went into the kernel over the last year as well as the development processes. The Linux kernel is probably one of the largest open source projects, and very healthy - there are a lot of individuals and companies contributing to it. With this size comes a few interesting challenges coping with the flow of changes into Linus' tree. Their current processes seem to deal with this quite well, and don't seem to need a lot of major changes at the moment.

His talk also included the obligatory list of features that landed in the last year. The only one that really matters to me is the Nouveau driver, which I'm looking forward to trying out.

The second talk I went to in the morning was Selena Deckelmann's overview of the Open Source database landscape. She mentioned there's new projects started daily, but it was still a bit disappointing not to see TDB up there.

After lunch Rob gave a talk about Subunit, introducing to the ideas behind the Subunit protocol as well as presenting an overview of the tools that are available for it and the projects that have Subunitized as of yet. It's exciting to see the Subunit universe slowly growing, I wasn't aware of some of the projects that are using it. The recently announced testrepository also looks interesting, even though it is still very rudimentary at the moment.

In the evening Tridge, Rusty, Andrew, Jeremy, AJ and I participated in the hackoff as the "Samba Team".

The hackoff was a lot of fun, and consisted of 6 problems, each of which involved somehow decoding the data file for the problem and extracting a short token from it in one way or another, which was required to retrieve the next problem. We managed to solve 4 problems in the hour that the organizers had allocated, and ended first because we were a bit quicker in solving the 4th problem than the runner-ups. No doubt the fact that we were the largest team had something to do with this.

I hung out with some of the awesome Git and Github developers in the Malthouse in the evening, and talked about Dulwich, Bazaar and Launchpad ("No really, I am not aware of any plans to add Git support to Launchpad.").

comments.

Linux.Conf.Au 2010 - Day 2 - Tuesday

On Tuesday we had our "Launchpad" mini-conf, which featured talks from Launchpad developers and users alike. It wasn't necessarily about hosting projects on Launchpad, but rather about how various projects could benefit from Launchpad.

I popped out of the Launchpad track for a bit to attend Andrews talk about the current status of Samba 4. He did a nice job of summarizing the events in the last year, the most of import one of course being the support for DC synchronization. I'm proud we've finally managed to pull this off - and hopefully we'll actually have a beta out next year. We have been saying "maybe next year" for almost 4 years now when people asked us for estimates of a release date.

At the end of the day Aaron and I gave a quick introduction to Launchpad's code imports and code reviews.

comments.