Posts by 'Jelmer Vernooij' – Page 10

1.0… at last?

Doing a bit of BitlBee development again during the last few days. Wilmer released 0.93 yesterday, and we’ve hopefully now also tracked down the bloody 100% CPU usage bug we have been chasing for the past 6 months. That’s the last bit on the roadmap to the big 1.0 release.

I’ve also started on rewriting the Windows port of BitlBee as a service rather then as a user program. The current version was rather unstable, mostly because it had to deal with a GUI thread and BitlBee’s core thread which is not really written with threads in mind.

The Win32 branch is available at win32.bitlbee.org.

comments.

GSSAPI patch for Dovecot

My GSSAPI patch for Dovecot is now in CVS. Thanks to Timo for committing it to CVS, and to pod for fixing some issues. This patch just implements authentication.

We’re now going to work on support for integration verification (sign) and protection (seal).

comments.

Results published by Google

Google has posted a summary of the results of the Summer of Code on the Company Blog. There’s also a link in there to a cool map with all the participants and mentors.

Update 23:13 Slashdot now also has a story.

comments.

scons issues

While I was very optimistic about scons a month ago, I must admit that enthusiasm has faded a bit now that we tried out to use it for building Samba4. The idea behind it is good, but there are some issues when using it in real life (not all of them huge surprises):

  • It assumes the relations between subsystems are compositional. If dir a is processed before dir b, dir a can not depend on any object files from dir b. This is a major problem for Samba.
  • Running the configure tests (even with caching) each time you run a build takes way too much time for a project with as much tests as Samba (althought there are hacks around it).
  • No easy way to add configure test functions.
  • Python. This is a problem for Samba both because it means an extra dependency and not a lot of committers are familiar with it.
  • Slow. Calculating checksums for each file rather then relying on modification times may be nice for those that have their sources on NFS, but it slows scons down a lot.
  • Very much focussed on files and conversions between files - generating a list of init functions like we do at the moment is quite hard.
  • While in theory it should be possible to generate a configure script and Makefile.in from a set of SConscript files, this proved to be a bit harder then I’d hoped.

Of course, it’s not all bad. Some good things about scons:

  • Can transparently handle portability issues with different compilers, architectures
  • The support for multiple build types is very neat and useful
  • Automatic support for header dependencies
  • Easy to extend. Scanning for dependencies, adding file types, adding compilers is not hard.
  • Compositional to a large degree. Being able to put the list of files and the list of functions/headers a subsystem needs in one file in that subsystems dir is very useful.

I’m not really sure where to go to from here. I am no longer convinced scons is the right way to go, though using some of the ideas from scons in the current build system might be good.

comments.

Bazaar-ng

Over the past few days I’ve been working on random bits of Bazaar-ng. Lots of interesting things are happening in the bzr world.

I’m now working on a plugin that adds support for Subversion branches.

comments.

Pidl and compatibility #2

Apparently MIDL does support something that is equivalent subcontext() in pidl. It has two attributes, represent_as, and transmit_as, which do something similar and are just about what I need for subcontext()s replacement. Looks like the only thing left to figure out now is what the arguments to the conversion functions will be.

comments.

Pidl and compatibility

One of the areas of pidl I am currently to improve is compatibily with other IDL compilers, so that exchanging IDL files becomes easier. We currently have extensions in pidl (with good reason!), that are unfortunately not supported by other compilers. My aim is therefore to not make support for these extensions a requirement for being able to load IDL files that use them. The easiest way to support something like that is by removing a few of the pidl-specific attributes before compiling with another compiler.

A recent change I made to pidl was the addition of support for the [string] attribute that is traditionally used in DCE/RPC IDL to make strings (which are nothing more then zero-terminated arrays, as far as NDR is concerned). For example, a string can be declared as follows in traditional IDL:

1
[string]  wchar *bla;

pidl previously had it’s own way of handling strings (a “string” type with special flags to indicate whether it was in ASCII, unicode, fixed-length, etc) which had the advantage that it would take care of handling the conversion between whatever character encoding was used on the wire and whatever one was used by Samba internally. The string specified above would look something like this:

1
[flag(STR_SIZE4|STR_LEN4)] string *bla;

In order to support [string] and also do on-the-fly character set conversions, I also added a new [charset()] attribute that tells Samba from what character set to convert. The string above would now look something like this:

1
[string,charset(UTF16)] wchar *bla;

One of the other extensions that needs to be a bit more optional is the [subcontext] attribute. This attribute is currently used for interpreting various generic data blobs that are send using DCE/RPC and traditionally interpreted by another layer in Windows. Subcontexts allow us to merge these two layers in Samba. For example the following code on windows:

1
2
uint32 size;
[size_is(size)] uint8 data[];

(data would then by casted to a FOO_BAR on a higher level), would look like this in Samba at the moment:

1
2
uint32 size;
[subcontext(4)] FOO_BAR data;

The problem here is that [subcontext()] is not very flexible. It’s argument is not very intuitive to use, and there are two other attributes available that change its behaviour.

I’d like to, in the future, replace subcontext with something like this:

1
2
uint32 size;
[size_is(size),replace(FOO_BAR)] uint8 data[];

Pidl would then call the push / pull / print functions of FOO_BAR after processing data[]. The uint8 array would only be declared locally in the pull / push / print function and “FOO_BAR data;” would appear in the public structures.

I’m not yet clear on what the semantics of the pull / print / push functions for FOO_BAR would be. Requiring names like ndr_{pull,push,print}_FOOBAR and the standard NDR arguments would certainly make sense, but it would also be great if we could support custom non-NDR subcontexts more easily (such as TDR-generated or hand-written ones). A possible solution would be allowing something like [replace(FOO_BAR,tdr)], maybe having the prefix default to “ndr”?

comments.

pidl-generated code now in Samba3

The first autogenerated RPC interface (DFS) is now in SVN, thanks to Jeremy. Hopefully the rest will follow soon.

comments.

Samba3 backend to pidl

Two days ago, I finally took the time to sit down and attempt to write a Samba3 backend to pidl (the IDL compiler in Samba 4). Turned out to be much easier then I expected! It took me two evenings (~16 hours total) to write basic NDR parser, header, server, client and template generators.

The code currently generated runs inside of the Samba3 tree without modifications (uses no additional helper functions and the like). I’m currently focussing on making the DFS interface autogenerated. Most things seem to work now, except for some name changes (the functions in Samba3 and Samba4’s IDL don’t have the same names). I’m working on that now. The latest generated sources are available at http://samba.org/~jelmer/pidl_samba3/ .

comments.

NUnit in unstable

My first package (nunit) is now in the Debian archive. It was approved by ftp-master two days ago.

Thanks go to Andrew “ajmitch” Mitchell for sponsoring and to Mirco “meebey” Bauer and David “Dajobe” Beckett for comments and feedback. See the pkg-mono pages for details.

Meebey has uploaded another package (gnunit) that provides a Gtk# interface to NUnit. Screenshot

comments.

Bazaar-ng and tailor

Over the last few days, I’ve done another attempt to generate a Bazaar-NG version of Samba4 using tailor. I had to fix a couple of small bugs but got it working eventually. It took me about 8 hours for 5924 revisions using the following tailor configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#!/usr/bin/env tailor

"""
[DEFAULT]
verbose = Yes

[samba4]
source = svn:samba4
target = bzrng:samba4
refill-changelogs = Yes
state-file = samba4.state
root-directory = ~/bzr/bzr.samba4
before-commit = samba_remap_authors

[svn:samba4]
repository = svn://svnanon.samba.org/samba
module = /branches/SAMBA_4_0
use-propset = Yes

[bzrng:samba4]
repository = ~/bzr/samba4
"""


def samba_remap_authors(context, changeset):
    authors_map = {
       u'metze': u'Stefan Metzmacher',
       u'tridge': u'Andrew Tridgell',
       u'abartlet': u'Andrew Bartlett',
       u'tpot': u'Tim Potter',
       u'jerry': u'Gerald (Jerry) Carter',
       u'vlendec': u'Volker Lendecke',
       u'crh': u'Chris Hertel',
       u'jra': u'Jeremy Allison',
       u'jelmer': u'Jelmer Vernooij',
       u'vance': u'Vance Lankhaar',
       u'derell': u'Derell Lipman',
       u'gd': u'Guenther Deschner',
       u'lmuelle': u'Lars Mueller',
       u'herb': u'Herb Lewis',
       u'jht': u'John Terpstra',
       u'jmcd': u'Jim McDonough',
       u'stevef': u'Steve French',
       u'jpeach': u'James Peach',
       u'mimir': u'Rafal Szczesniak',
       u'ab': u'Alexander Bokovoy',
       u'deryck': u'Deryck Hodge',
       u'lha': u'Love H\uffffrnquist \uffffstrand',
       u'idra': u'Simo Sorce',
       u'rsharpe': u'Richard Sharpe'
    }
    author = changeset.author
    if authors_map.has_key(author):
        changeset.author = "%s <%s@samba.org>" % (authors_map[author],author)
    return True

This branch is now up on samba.org. You’ll need the bzr “newformat” branch in order to check it out.

I’m a bit disappointed by the speed of “bzr branch” (takes around 3 hours when run locally, pulling in those ~6000 revisions).

Branching from a remote branch still gives me an error (this branch is up at http://samba.org/~jelmer/bzr.samba4/ and is the same I pull from locally):

1
2
3
bzr: ERROR: 'RemoteBranch' object has no attribute 'weave_store'
  at /usr/lib/python2.4/site-packages/bzrlib/fetch.py line 92, in __init__()
  see ~/.bzr.log for debug information

comments.

Pidl now in CPAN

Encouraged by the fact that ethereal now uses pidl as part of its’ standard build, I tried to upload Pidl as a package to CPAN today. Turned out to be much easier to become a perl developer then to become a Debian developer… :-)

More information is available at Pidl at CPAN.

comments.

Version Control Systems

It’s been a while since I’ve written about version control and the landscape has changed quite a bit since then. Writing your own SCM seems to be the latest hype…

I’m currently using Subversion for most of the projects I work on (including those on Windows). Several other systems look interesting. I especially like darcs, as it is written in Haskell and has an elegant UI, though it is horribly slow. Bzr looks very promising, though its doesn’t really store data efficiently at the moment.

Another very interesting project, and a very useful one now with all the various SCM’s around, is tailor - a tool that can do conversions between the various projects.

comments.

SoC Deadline

So, just a few more minutes until the deadline of the Google Summer of Code, 2005! I got pretty much all that I wanted in there done, except for the ldb_map module, which is still a bit dodgy.

Participating was a nice and interesting experience! The other participants that were working on Samba for SoC have produced some great code and it looks like they’re going to stick around. Especially Brads’ work on DSSYNC looks promising. I can certainly recommend anyone to participate if Google is repeating SoC again sometime in the future!

Looking back, I guess it could’ve been interesting to work on a different project then Samba such as mono (I do actually actively follow mono development, I just don’t contribute at the moment), althought what I did now was probably more productive and allowed me to spend some time on other sides of Samba than I am usually involved in as well.

I just committed the file STATUS file to the tree:

—- Samba3 -> Samba4 Upgrade —- (C) 2005 Jelmer Vernooij <jelmer@samba.org> Published under the GNU GPL

Sponsored by the Google Summer of Code program (http://code.google.com/summerofcode.html) Mentored by Andrew Bartlett <abartlet@samba.org> Thanks!

Done:
  • Reading wins.dat
  • Reading registry.tdb
  • Reading passdb.tdb
  • Reading account_policy.tdb
  • Reading group_mappings.tdb
  • Reading winbindd_idmap.tdb
  • Reading share_info.tdb
  • Reading secrets.tdb
  • Reading smbpasswd
  • Reading + writing (generic) smb.conf files
  • Testsuite for read support mentioned above
  • Console utility for dumping Samba information
  • Import user accounts in Samba4
  • Import groups in Samba4
  • Import secrets in Samba4
  • Import WINS data in Samba4
  • Dump idmap data to LDB
  • Import registry keys/values in Samba4
  • Import account policies in Samba4
  • Testsuite for upgrade
  • Console utility from upgrading from Samba3 -> Samba4
  • SWAT (Web interface) support for upgrading from Samba3 -> Samba4
  • LDB generic mapping module
  • (Experimental) Samba4 LDB <-> Samba3 LDAP mapping module based on LDB generic mapping module
  • Testsuite for Samba4 LDB <-> Samba3 LDAP mapping module

Source files: source/lib/ldb/modules/ldb_map.c source/lib/ldb/modules/ldb_map.h source/lib/samba3/group.c source/lib/samba3/idmap.c source/lib/samba3/policy.c source/lib/samba3/registry.c source/lib/samba3/samba3.c source/lib/samba3/secrets.c source/lib/samba3/share_info.c source/lib/samba3/smbpasswd.c source/lib/samba3/tdbsam.c source/lib/samba3/winsdb.c source/lib/samba3/samba3.h source/scripting/libjs/upgrade.js source/scripting/ejs/smbcalls_param.c source/scripting/ejs/smbcalls_samba3.c source/param/generic.c source/param/generic.h testdata/samba3/verify testprogs/ejs/samba3sam source/setup/upgrade source/scripting/bin/samba3dump source/dsdb/samdb/ldb_modules/samba3sam.c source/script/tests/test_s3upgrade.sh swat/install/samba3.esp

Known remaining issues:
  • [upgrade] Conversion from the smbpasswd/TDB passwords to ntPwdHash / lmPwdHash is broken. Couldn’t find out why.
  • [ldb_map] Conversion of attribute names in DN’s is still a bit dodgy
  • [ldb_map] mapped objectClass names may be mentioned multiple times in returned records
  • [ldb_map] add/modify support not tested very well with LDAP yet (only LDB+TDB)
  • [ldb_map] group membership is not yet mapped (only primaryGroupID / sambaPrimaryGroupSID)

comments.

The alternative approach

Looking more closely at one of the other approaches I could’ve taken for migrating from Samba3. We will probably end up supporting this method later on anyway, once we expand the vampire code. I do think the path I’ve taken for SoC is the right one though - it allows the upgrade with the least hassle and user input.

This approach could be splitted up into the various parts:

  • samba3 pidl backend
  • Samba4 vampire + server side samsync support in Samba3
  • unixinfo (unixinfo) (exposes idmap) - in Samba4 (client side) - in Samba3 (server side)
  • winsrepl (thru seperate pipe?)
  • enum/add shares (srvsvc)
  • enum/add registry (winreg)
  • enum/add printers (spoolss(?))
  • convert smb.conf (using Jerry’s registry hack)

Advantages of this approach:

  • Can be used to ‘clone’ Samba4 boxes as well
  • Relatively few extra loc necessary (a lot shared with vampire)
  • ‘Clean’ code - simply enumerates and adds

Disadvantages:

  • Needs running Samba3 daemon for upgrade (!)
  • Needs (complex?) Samba3 code changes
  • Needs latest version of Samba3

comments.