It took us almost 3.5 years, but at last we managed to reach the important milestone: Bitlbee 1.0 is now officially released! We held a small release party in Utrecht this afternoon, including a formal release speech from Sjoerd (a blessed Bitlbee tradition) and pizza. I took some photos.
New challenges await us now as we continue to cleanup the various protocol implementations and start working on 1.1 and 2.0.
Today I spent some time working again on foreign branch support for bzr. I got quite far, and can now run ‘bzr log’ and ‘bzr viz’ on remote Subversion branches.
The big hurdle to take were the Python bindings for Subversion, which are a pain to work with. Because these bindings were generated with SWIG (and haven’t had enough customization by humans), they very much look like C functions that can be called from python. The programmer has to worry about memory management, some functions with callback methods can not be used and passing an incorrect parameter usually results in a segmentation fault.
Most commands appear to run very slowly, but I found out this was caused, while I had initially blamed bzr, by ‘svn ls’ being extremely slow. It takens up to a few minutes to get a list of all files in a repository. This is of course a big problem if you’re trying to generate 2000 inventory objects, one for each revision….
While trying to work on foreign branch support today, I got distracted and hacked up a small post-commit hook for bzr that submits commits to CIA. Doing stuff like this is trivial to do with bzr if you know some python.
Wilmer announced the switch of Bitlbee’s source control system to bzr yesterday, so it’s finally official. This will hopefully make it easier for Maurits and me to contribute, and increase the number of external contributors.
Since we’ll be using trac as the bug tracking system for BitlBee and bzr for source control, we won’t be able to use the default post-commit-message-hook, which comes with trac. This script usually makes sure that whenever a commit is done that references a bug, this gets mentioned on the page of that particular bug in trac.
I’ve hacked up a quick plugin for bzr which implements the same thing for the combination of bzr and trac. This isn’t the ideal solution since it requires that the script is running on the same host as the trac server, which is sometimes not the case as people push changes to their server rather then log in to their server and merge and commit the changes from their workstation.
The proper fix is probably getting trac itself to get information like this from the commit messages. It already detects new commits (they are shown in Timeline) and knows how to parse them, so something like that shouldn’t be very hard to implement.
After three years without, it looks like BitlBee is finally going to leave the dark ages and use a version control system that was not thought up in Wilmers’ basement. Up until now, Wilmer kept a master copy of the tree from which he generated nightly snapshots. The other two developers (Maurits and me) would then send patches against those nightly snapshots which Wilmer would apply. This sort of worked, but it wasn’t ideal: migrating in-progress patches from snapshot to snapshot takes a lot of time and creating incremental patches can be impractical as well.
Some of the objections Wilmer had against using a version control system, in particular CVS and Subversion (which I was originally trying to sell him) were the fact that they require a manual svn add for all files you create, a svn rm for files you delete and the requirement for setting up a special server. At the moment, we’re experimenting with bzr since it doesn’t have these limitations.
We’re probably going with trac+darcs for the new bug tracking system.
Doing a bit of BitlBee development again during the last few days. Wilmer released 0.93 yesterday, and we’ve hopefully now also tracked down the bloody 100% CPU usage bug we have been chasing for the past 6 months. That’s the last bit on the roadmap to the big 1.0 release.
I’ve also started on rewriting the Windows port of BitlBee as a service rather then as a user program. The current version was rather unstable, mostly because it had to deal with a GUI thread and BitlBee’s core thread which is not really written with threads in mind.
The Win32 branch is available at win32.bitlbee.org.
My GSSAPI patch for Dovecot is now in CVS. Thanks to Timo for committing it to CVS, and to pod for fixing some issues. This patch just implements authentication.
We’re now going to work on support for integration verification (sign) and protection (seal).
Google has posted a summary of the results of the Summer of Code on the Company Blog. There’s also a link in there to a cool map with all the participants and mentors.
While I was very optimistic about scons a month ago, I must admit that enthusiasm has faded a bit now that we tried out to use it for building Samba4. The idea behind it is good, but there are some issues when using it in real life (not all of them huge surprises):
Of course, it’s not all bad. Some good things about scons:
I’m not really sure where to go to from here. I am no longer convinced scons is the right way to go, though using some of the ideas from scons in the current build system might be good.
Over the past few days I’ve been working on random bits of Bazaar-ng. Lots of interesting things are happening in the bzr world.
I’m now working on a plugin that adds support for Subversion branches.
Apparently MIDL does support something that is equivalent subcontext() in pidl. It has two attributes, represent_as, and transmit_as, which do something similar and are just about what I need for subcontext()s replacement. Looks like the only thing left to figure out now is what the arguments to the conversion functions will be.
One of the areas of pidl I am currently to improve is compatibily with other IDL compilers, so that exchanging IDL files becomes easier. We currently have extensions in pidl (with good reason!), that are unfortunately not supported by other compilers. My aim is therefore to not make support for these extensions a requirement for being able to load IDL files that use them. The easiest way to support something like that is by removing a few of the pidl-specific attributes before compiling with another compiler.
A recent change I made to pidl was the addition of support for the [string] attribute that is traditionally used in DCE/RPC IDL to make strings (which are nothing more then zero-terminated arrays, as far as NDR is concerned). For example, a string can be declared as follows in traditional IDL:
1 | [string] wchar *bla; |
pidl previously had it’s own way of handling strings (a “string” type with special flags to indicate whether it was in ASCII, unicode, fixed-length, etc) which had the advantage that it would take care of handling the conversion between whatever character encoding was used on the wire and whatever one was used by Samba internally. The string specified above would look something like this:
1 | [flag(STR_SIZE4|STR_LEN4)] string *bla; |
In order to support [string] and also do on-the-fly character set conversions, I also added a new [charset()] attribute that tells Samba from what character set to convert. The string above would now look something like this:
1 | [string,charset(UTF16)] wchar *bla; |
One of the other extensions that needs to be a bit more optional is the [subcontext] attribute. This attribute is currently used for interpreting various generic data blobs that are send using DCE/RPC and traditionally interpreted by another layer in Windows. Subcontexts allow us to merge these two layers in Samba. For example the following code on windows:
1 2 | uint32 size; [size_is(size)] uint8 data[]; |
(data would then by casted to a FOO_BAR on a higher level), would look like this in Samba at the moment:
1 2 | uint32 size; [subcontext(4)] FOO_BAR data; |
The problem here is that [subcontext()] is not very flexible. It’s argument is not very intuitive to use, and there are two other attributes available that change its behaviour.
I’d like to, in the future, replace subcontext with something like this:
1 2 | uint32 size; [size_is(size),replace(FOO_BAR)] uint8 data[]; |
Pidl would then call the push / pull / print functions of FOO_BAR after processing data[]. The uint8 array would only be declared locally in the pull / push / print function and “FOO_BAR data;” would appear in the public structures.
I’m not yet clear on what the semantics of the pull / print / push functions for FOO_BAR would be. Requiring names like ndr_{pull,push,print}_FOOBAR and the standard NDR arguments would certainly make sense, but it would also be great if we could support custom non-NDR subcontexts more easily (such as TDR-generated or hand-written ones). A possible solution would be allowing something like [replace(FOO_BAR,tdr)], maybe having the prefix default to “ndr”?
The first autogenerated RPC interface (DFS) is now in SVN, thanks to Jeremy. Hopefully the rest will follow soon.
Two days ago, I finally took the time to sit down and attempt to write a Samba3 backend to pidl (the IDL compiler in Samba 4). Turned out to be much easier then I expected! It took me two evenings (~16 hours total) to write basic NDR parser, header, server, client and template generators.
The code currently generated runs inside of the Samba3 tree without modifications (uses no additional helper functions and the like). I’m currently focussing on making the DFS interface autogenerated. Most things seem to work now, except for some name changes (the functions in Samba3 and Samba4’s IDL don’t have the same names). I’m working on that now. The latest generated sources are available at http://samba.org/~jelmer/pidl_samba3/ .
