Jelmer Vernooĳ

NUnit in unstable

meta = {'category': debian, 'date': Tue 04 October 2005}

My first package (nunit) is now in the Debian archive. It was approved by ftp-master two days ago.

Thanks go to Andrew “ajmitch” Mitchell for sponsoring and to Mirco “meebey” Bauer and David “Dajobe” Beckett for comments and feedback. See the pkg-mono pages for details.

Meebey has uploaded another package (gnunit) that provides a Gtk# interface to NUnit. Screenshot

comments.

Bazaar-ng and tailor

meta = {'category': bazaar, 'date': Sun 25 September 2005}

Over the last few days, I’ve done another attempt to generate a Bazaar-NG version of Samba4 using tailor. I had to fix a couple of small bugs but got it working eventually. It took me about 8 hours for 5924 revisions using the following tailor configuration:

#!/usr/bin/env tailor

"""
[DEFAULT]
verbose = Yes

[samba4]
source = svn:samba4
target = bzrng:samba4
refill-changelogs = Yes
state-file = samba4.state
root-directory = ~/bzr/bzr.samba4
before-commit = samba_remap_authors

[svn:samba4]
repository = svn://svnanon.samba.org/samba
module = /branches/SAMBA_4_0
use-propset = Yes

[bzrng:samba4]
repository = ~/bzr/samba4
"""


def samba_remap_authors(context, changeset):
    authors_map = {
       u'metze': u'Stefan Metzmacher',
       u'tridge': u'Andrew Tridgell',
       u'abartlet': u'Andrew Bartlett',
       u'tpot': u'Tim Potter',
       u'jerry': u'Gerald (Jerry) Carter',
       u'vlendec': u'Volker Lendecke',
       u'crh': u'Chris Hertel',
       u'jra': u'Jeremy Allison',
       u'jelmer': u'Jelmer Vernooĳ',
       u'vance': u'Vance Lankhaar',
       u'derell': u'Derell Lipman',
       u'gd': u'Guenther Deschner',
       u'lmuelle': u'Lars Mueller',
       u'herb': u'Herb Lewis',
       u'jht': u'John Terpstra',
       u'jmcd': u'Jim McDonough',
       u'stevef': u'Steve French',
       u'jpeach': u'James Peach',
       u'mimir': u'Rafal Szczesniak',
       u'ab': u'Alexander Bokovoy',
       u'deryck': u'Deryck Hodge',
       u'lha': u'Love H\uffffrnquist \uffffstrand',
       u'idra': u'Simo Sorce',
       u'rsharpe': u'Richard Sharpe'
    }
    author = changeset.author
    if authors_map.has_key(author):
        changeset.author = "%s <%s@samba.org>" % (authors_map[author],author)
    return True

This branch is now up on samba.org. You’ll need the bzr “newformat” branch in order to check it out.

I’m a bit disappointed by the speed of “bzr branch” (takes around 3 hours when run locally, pulling in those ~6000 revisions).

Branching from a remote branch still gives me an error (this branch is up at http://samba.org/~jelmer/bzr.samba4/ and is the same I pull from locally):

bzr: ERROR: 'RemoteBranch' object has no attribute 'weave_store'
  at /usr/lib/python2.4/site-packages/bzrlib/fetch.py line 92, in __init__()
  see ~/.bzr.log for debug information

comments.

Pidl now in CPAN

meta = {'category': samba, 'date': Wed 21 September 2005}

Encouraged by the fact that ethereal now uses pidl as part of its’ standard build, I tried to upload Pidl as a package to CPAN today. Turned out to be much easier to become a perl developer then to become a Debian developer… :-)

More information is available at Pidl at CPAN.

comments.

Version Control Systems

meta = {'category': bazaar, 'date': Tue 06 September 2005}

It’s been a while since I’ve written about version control and the landscape has changed quite a bit since then. Writing your own SCM seems to be the latest hype…

I’m currently using Subversion for most of the projects I work on (including those on Windows). Several other systems look interesting. I especially like darcs, as it is written in Haskell and has an elegant UI, though it is horribly slow. Bzr looks very promising, though its doesn’t really store data efficiently at the moment.

Another very interesting project, and a very useful one now with all the various SCM’s around, is tailor - a tool that can do conversions between the various projects.

comments.

SoC Deadline

meta = {'category': samba, 'date': Thu 01 September 2005}

So, just a few more minutes until the deadline of the Google Summer of Code, 2005! I got pretty much all that I wanted in there done, except for the ldb_map module, which is still a bit dodgy.

Participating was a nice and interesting experience! The other participants that were working on Samba for SoC have produced some great code and it looks like they’re going to stick around. Especially Brads’ work on DSSYNC looks promising. I can certainly recommend anyone to participate if Google is repeating SoC again sometime in the future!

Looking back, I guess it could’ve been interesting to work on a different project then Samba such as mono (I do actually actively follow mono development, I just don’t contribute at the moment), althought what I did now was probably more productive and allowed me to spend some time on other sides of Samba than I am usually involved in as well.

I just committed the file STATUS file to the tree:

Sponsored by the Google Summer of Code program (http://code.google.com/summerofcode.html) Mentored by Andrew Bartlett <abartlet@samba.org> Thanks!

Done:

Reading wins.dat
Reading registry.tdb
Reading passdb.tdb
Reading account_policy.tdb
Reading group_mappings.tdb
Reading winbindd_idmap.tdb
Reading share_info.tdb
Reading secrets.tdb
Reading smbpasswd
Reading + writing (generic) smb.conf files
Testsuite for read support mentioned above
Console utility for dumping Samba information
Import user accounts in Samba4
Import groups in Samba4
Import secrets in Samba4
Import WINS data in Samba4
Dump idmap data to LDB
Import registry keys/values in Samba4
Import account policies in Samba4
Testsuite for upgrade
Console utility from upgrading from Samba3 -> Samba4
SWAT (Web interface) support for upgrading from Samba3 -> Samba4
LDB generic mapping module
(Experimental) Samba4 LDB <-> Samba3 LDAP mapping module based on LDB generic mapping module
Testsuite for Samba4 LDB <-> Samba3 LDAP mapping module

Source files: source/lib/ldb/modules/ldb_map.c source/lib/ldb/modules/ldb_map.h source/lib/samba3/group.c source/lib/samba3/idmap.c source/lib/samba3/policy.c source/lib/samba3/registry.c source/lib/samba3/samba3.c source/lib/samba3/secrets.c source/lib/samba3/share_info.c source/lib/samba3/smbpasswd.c source/lib/samba3/tdbsam.c source/lib/samba3/winsdb.c source/lib/samba3/samba3.h source/scripting/libjs/upgrade.js source/scripting/ejs/smbcalls_param.c source/scripting/ejs/smbcalls_samba3.c source/param/generic.c source/param/generic.h testdata/samba3/verify testprogs/ejs/samba3sam source/setup/upgrade source/scripting/bin/samba3dump source/dsdb/samdb/ldb_modules/samba3sam.c source/script/tests/test_s3upgrade.sh swat/install/samba3.esp

Known remaining issues:

[upgrade] Conversion from the smbpasswd/TDB passwords to ntPwdHash / lmPwdHash is broken. Couldn’t find out why.
[ldb_map] Conversion of attribute names in DN’s is still a bit dodgy
[ldb_map] mapped objectClass names may be mentioned multiple times in returned records
[ldb_map] add/modify support not tested very well with LDAP yet (only LDB+TDB)
[ldb_map] group membership is not yet mapped (only primaryGroupID / sambaPrimaryGroupSID)

comments.

The alternative approach

meta = {'category': samba, 'date': Mon 29 August 2005}

Looking more closely at one of the other approaches I could’ve taken for migrating from Samba3. We will probably end up supporting this method later on anyway, once we expand the vampire code. I do think the path I’ve taken for SoC is the right one though - it allows the upgrade with the least hassle and user input.

This approach could be splitted up into the various parts:

samba3 pidl backend
Samba4 vampire + server side samsync support in Samba3
unixinfo (unixinfo) (exposes idmap) - in Samba4 (client side) - in Samba3 (server side)
winsrepl (thru seperate pipe?)
enum/add shares (srvsvc)
enum/add registry (winreg)
enum/add printers (spoolss(?))
convert smb.conf (using Jerry’s registry hack)

Advantages of this approach:

Can be used to ‘clone’ Samba4 boxes as well
Relatively few extra loc necessary (a lot shared with vampire)
‘Clean’ code - simply enumerates and adds

Disadvantages:

Needs running Samba3 daemon for upgrade (!)
Needs (complex?) Samba3 code changes
Needs latest version of Samba3

comments.

Partially working upgrade paths

meta = {'category': bazaar, 'date': Mon 29 August 2005}

The one-way upgrade from Samba3 to Samba4 is now partially working! There are both a web-frontend and a console utility that can do an upgrade, given a libdir (usually /var/lib/samba) and a configuration file (usually in /etc/samba/smb.conf).

Last night, I rewrote most of the conversion code to be in EJS rather then in C. One of the positive things about this is that I can use embedded LDIF inside of the JavaScript code and that I can build on the existing Samba4 provisioning code (which is in EJS as well). Writing EJS wrappers for my existing C code was easier then I had thought.

One of the other things I did today is add a fallback mechanism for ldb_map where it can store the data that could not be mapped. There are now two types of records in ldb_map: those that are partially mapped (mapped, but might have one or two attributes stored in the fallback database) and those that are completely stored in the fallback database.

Below are some screenshots of the upgrade procedure in SWAT: /blog/uploads/screenshots/upgr1.png /blog/uploads/screenshots/upgr2.png /blog/uploads/screenshots/upgr3.png

comments.

LDIF output working

meta = {'category': samba, 'date': Sun 28 August 2005}

The sam.ldif and winbind.ldif generation code now works as well. I’ll be working on generating the new smb.conf next.

One of the things that worries me is the fact that I can generate so few of the Samba4 ‘user’ and ‘group’ attributes. Probably need to borrow some code from the current provisioning script to fix that up.

I’m wondering whether I should do some of the upgrade using EJS, so it can be integrated into SWAT easily…

comments.

ldb_map somewhat working

meta = {'category': samba, 'date': Sun 28 August 2005}

ldb_map and the samba3sam ldb module that is based on it work nicely now and appear to be memory-leak free. I have so far been testing with a TDB backend loaded with an LDIF from Samba3. Tomorrow will be the big day, testing against an OpenLDAP server.

Since the LDAP schema for Samba3 contains information that the Samba4 SAM database doesn’t store and vice versa, it will be interesting to find out whether the current mapping will be sufficient. If it is not, I might have to fall back to storing the Samba4 attributes I can’t store in the Samba3 LDAP database in a seperate database instead.

Another thing I did today is clean up the parameter list in Samba4 - I removed all unused parameters. This should hopefully make it easier to do the Samba3->Samba4 smb.conf conversion code.

TODO is down to:

smb.conf conversion
Finish sam.ldif / winbind.ldif from the TDB files
Test samba3sam module against OpenLDAP

comments.

First LDIF working

meta = {'category': samba, 'date': Thu 25 August 2005}

Today was a productive day. I just got the LDIF output for samba3dump working for wins.ldif and the various registry hives. TODO is down to:

sam.ldif
winbind.ldif
smb.conf
test and finish ldb_map and ldb_samba3

I hope to get the first three done tomorrow.

comments.

Starting on LDIF output

meta = {'category': samba, 'date': Thu 25 August 2005}

Next on my list is the generation of the LDIF’s for the one-way upgrade path. These will be generated by internally building ldb_ldif structs and then dumping them using the LDB helper functions. This would easily allow us to later on directly write to LDB database.

I’ll need to generate the following LDIF files:

hklm / hkcu / etc.ldif: from samba3_regdb
wins.ldif: from samba3_winsdb
sam.ldif: from samba3_secrets, samba3_policy, samba3_privileges, samba3_samdb
winbind.ldif: from samba3_winbind

I’ll also make a start on the smb.conf output.

comments.

ldb_map done

meta = {'category': samba, 'date': Thu 25 August 2005}

Finally got the ldb_map code done. This is a simple skeleton module that takes a mapping table of LDAP attributes and objectclasses and then maps between two different schema’s. Also got the initial part of ldb_samba3 compiling, which is the first module that actually uses ldb_map.

Yesterday evening, I also added support for reading smb.conf files and share_info.tdb files to the lib/samba3/ directory. Looks like my TODO-list is now down to:

testing (writing a couple of smbtorture tests, preferably)
code for generating new smb.conf and for generating LDIF from the various TDB’s

comments.

TDB reading done

meta = {'category': samba, 'date': Wed 24 August 2005}

I’ve now finished the ability to read the various TDB files. I’ve also started on the “mapping” LDB module that will allow a simple mapping from one objectclass to another. We’ll need this for the backwards compatible LDAP backend for Samba3.

This backend will never be used itself but instead have other backends based on it (e.g. ldb_samba3 for a Samba3-LDAP-to-samdb backend)

Each backend that uses this skeleton has to specify an array of struct objectclass_mapping’s (see ldb_map.h).

If no mappings are defined for an objectclass, it will be kept as is.

Will also keep cache of mapped dn’s so we don’t have to traverse the tree each time

So, TODO left:

finish ldb_map
fill ldb_samba3’s mapping table
support smb.conf parsing in lib/samba3/
support generating LDIF’s

comments.

Samba4 debian packages

meta = {'category': debian, 'date': Tue 23 August 2005}

Now that the Samba4 Debian packages appear to be somewhat working, I’ve set up a daily cronjob that generates packages for the latest SVN revision. Add the following line to your “/etc/apt/sources.list” to try them out:

1	deb http://jelmer.vernstok.nl/debian/autobuild .

Please note that these packages are still very much alpha, as is the Samba4 code. Comments are welcome

comments.

Status Update

meta = {'category': samba, 'date': Tue 23 August 2005}

My Summer of Code project is coming along nicely (although I haven’t been very actively blogging about it). I’ll be working on full-time until the deadline, I guess, but I think I’ll make it.

So far I’ve got the following done:

Reading of tdbsam
Reading of policy database (still need to add privilege support)
Reading of idmap database
Reading of registry file (still needs some work)
Reading of old smb.conf files
Reading of group mapping database
Initial work on ldb_map backend

The major lump of work now is the ldb_map backend and the conversion of the smb.conf.

comments.

Posts – Page 11