Posts in 'samba' – Page 3

New Samba4 Debian packages

Thanks to the help of Steinar Gunderson in the past week, the Debian packages of Samba4 have had some major improvements. I have once again started to provide snapshots. As usual, they can be used by adding the following line to your /etc/apt/sources.list:

deb ftp://jelmer.vernstok.nl/pub/debian ./

comments.

Results published by Google

Google has posted a summary of the results of the Summer of Code on the Company Blog. There's also a link in there to a cool map with all the participants and mentors.

Update 23:13 Slashdot now also has a story.

comments.

scons issues

While I was very optimistic about scons a month ago, I must admit that enthusiasm has faded a bit now that we tried out to use it for building Samba4. The idea behind it is good, but there are some issues when using it in real life (not all of them huge surprises):

  • It assumes the relations between subsystems are compositional. If dir a is processed before dir b, dir a can not depend on any object files from dir b. This is a major problem for Samba.
  • Running the configure tests (even with caching) each time you run a build takes way too much time for a project with as much tests as Samba (althought there are hacks around it).
  • No easy way to add configure test functions.
  • Python. This is a problem for Samba both because it means an extra dependency and not a lot of committers are familiar with it.
  • Slow. Calculating checksums for each file rather then relying on modification times may be nice for those that have their sources on NFS, but it slows scons down a lot.
  • Very much focussed on files and conversions between files - generating a list of init functions like we do at the moment is quite hard.
  • While in theory it should be possible to generate a configure script and Makefile.in from a set of SConscript files, this proved to be a bit harder then I'd hoped.

Of course, it's not all bad. Some good things about scons:

  • Can transparently handle portability issues with different compilers, architectures
  • The support for multiple build types is very neat and useful
  • Automatic support for header dependencies
  • Easy to extend. Scanning for dependencies, adding file types, adding compilers is not hard.
  • Compositional to a large degree. Being able to put the list of files and the list of functions/headers a subsystem needs in one file in that subsystems dir is very useful.

I'm not really sure where to go to from here. I am no longer convinced scons is the right way to go, though using some of the ideas from scons in the current build system might be good.

comments.

Pidl and compatibility #2

Apparently MIDL does support something that is equivalent subcontext() in pidl. It has two attributes, represent_as, and transmit_as, which do something similar and are just about what I need for subcontext()s replacement. Looks like the only thing left to figure out now is what the arguments to the conversion functions will be.

comments.

Pidl and compatibility

One of the areas of pidl I am currently to improve is compatibily with other IDL compilers, so that exchanging IDL files becomes easier. We currently have extensions in pidl (with good reason!), that are unfortunately not supported by other compilers. My aim is therefore to not make support for these extensions a requirement for being able to load IDL files that use them. The easiest way to support something like that is by removing a few of the pidl-specific attributes before compiling with another compiler.

A recent change I made to pidl was the addition of support for the [string] attribute that is traditionally used in DCE/RPC IDL to make strings (which are nothing more then zero-terminated arrays, as far as NDR is concerned). For example, a string can be declared as follows in traditional IDL:

[string]  wchar *bla;

pidl previously had it's own way of handling strings (a "string" type with special flags to indicate whether it was in ASCII, unicode, fixed-length, etc) which had the advantage that it would take care of handling the conversion between whatever character encoding was used on the wire and whatever one was used by Samba internally. The string specified above would look something like this:

[flag(STR_SIZE4|STR_LEN4)] string *bla;

In order to support [string] and also do on-the-fly character set conversions, I also added a new [charset()] attribute that tells Samba from what character set to convert. The string above would now look something like this:

[string,charset(UTF16)] wchar *bla;

One of the other extensions that needs to be a bit more optional is the [subcontext] attribute. This attribute is currently used for interpreting various generic data blobs that are send using DCE/RPC and traditionally interpreted by another layer in Windows. Subcontexts allow us to merge these two layers in Samba. For example the following code on windows:

uint32 size;
[size_is(size)] uint8 data[];

(data would then by casted to a FOO_BAR on a higher level), would look like this in Samba at the moment:

uint32 size;
[subcontext(4)] FOO_BAR data;

The problem here is that [subcontext()] is not very flexible. It's argument is not very intuitive to use, and there are two other attributes available that change its behaviour.

I'd like to, in the future, replace subcontext with something like this:

uint32 size;
[size_is(size),replace(FOO_BAR)] uint8 data[];

Pidl would then call the push / pull / print functions of FOO_BAR after processing data[]. The uint8 array would only be declared locally in the pull / push / print function and "FOO_BAR data;" would appear in the public structures.

I'm not yet clear on what the semantics of the pull / print / push functions for FOO_BAR would be. Requiring names like ndr_{pull,push,print}_FOOBAR and the standard NDR arguments would certainly make sense, but it would also be great if we could support custom non-NDR subcontexts more easily (such as TDR-generated or hand-written ones). A possible solution would be allowing something like [replace(FOO_BAR,tdr)], maybe having the prefix default to "ndr"?

comments.

pidl-generated code now in Samba3

The first autogenerated RPC interface (DFS) is now in SVN, thanks to Jeremy. Hopefully the rest will follow soon.

comments.

Samba3 backend to pidl

Two days ago, I finally took the time to sit down and attempt to write a Samba3 backend to pidl (the IDL compiler in Samba 4). Turned out to be much easier then I expected! It took me two evenings (~16 hours total) to write basic NDR parser, header, server, client and template generators.

The code currently generated runs inside of the Samba3 tree without modifications (uses no additional helper functions and the like). I'm currently focussing on making the DFS interface autogenerated. Most things seem to work now, except for some name changes (the functions in Samba3 and Samba4's IDL don't have the same names). I'm working on that now. The latest generated sources are available at http://samba.org/~jelmer/pidl_samba3/ .

comments.

Pidl now in CPAN

Encouraged by the fact that ethereal now uses pidl as part of its' standard build, I tried to upload Pidl as a package to CPAN today. Turned out to be much easier to become a perl developer then to become a Debian developer... :-)

More information is available at Pidl at CPAN.

comments.

SoC Deadline

So, just a few more minutes until the deadline of the Google Summer of Code, 2005! I got pretty much all that I wanted in there done, except for the ldb_map module, which is still a bit dodgy.

Participating was a nice and interesting experience! The other participants that were working on Samba for SoC have produced some great code and it looks like they're going to stick around. Especially Brads' work on DSSYNC looks promising. I can certainly recommend anyone to participate if Google is repeating SoC again sometime in the future!

Looking back, I guess it could've been interesting to work on a different project then Samba such as mono (I do actually actively follow mono development, I just don't contribute at the moment), althought what I did now was probably more productive and allowed me to spend some time on other sides of Samba than I am usually involved in as well.

I just committed the file STATUS file to the tree:

--- Samba3 -> Samba4 Upgrade --- (C) 2005 Jelmer Vernooij <jelmer@samba.org> Published under the GNU GPL

Sponsored by the Google Summer of Code program (http://code.google.com/summerofcode.html) Mentored by Andrew Bartlett <abartlet@samba.org> Thanks!

Done:
  • Reading wins.dat
  • Reading registry.tdb
  • Reading passdb.tdb
  • Reading account_policy.tdb
  • Reading group_mappings.tdb
  • Reading winbindd_idmap.tdb
  • Reading share_info.tdb
  • Reading secrets.tdb
  • Reading smbpasswd
  • Reading + writing (generic) smb.conf files
  • Testsuite for read support mentioned above
  • Console utility for dumping Samba information
  • Import user accounts in Samba4
  • Import groups in Samba4
  • Import secrets in Samba4
  • Import WINS data in Samba4
  • Dump idmap data to LDB
  • Import registry keys/values in Samba4
  • Import account policies in Samba4
  • Testsuite for upgrade
  • Console utility from upgrading from Samba3 -> Samba4
  • SWAT (Web interface) support for upgrading from Samba3 -> Samba4
  • LDB generic mapping module
  • (Experimental) Samba4 LDB <-> Samba3 LDAP mapping module based on LDB generic mapping module
  • Testsuite for Samba4 LDB <-> Samba3 LDAP mapping module

Source files: source/lib/ldb/modules/ldb_map.c source/lib/ldb/modules/ldb_map.h source/lib/samba3/group.c source/lib/samba3/idmap.c source/lib/samba3/policy.c source/lib/samba3/registry.c source/lib/samba3/samba3.c source/lib/samba3/secrets.c source/lib/samba3/share_info.c source/lib/samba3/smbpasswd.c source/lib/samba3/tdbsam.c source/lib/samba3/winsdb.c source/lib/samba3/samba3.h source/scripting/libjs/upgrade.js source/scripting/ejs/smbcalls_param.c source/scripting/ejs/smbcalls_samba3.c source/param/generic.c source/param/generic.h testdata/samba3/verify testprogs/ejs/samba3sam source/setup/upgrade source/scripting/bin/samba3dump source/dsdb/samdb/ldb_modules/samba3sam.c source/script/tests/test_s3upgrade.sh swat/install/samba3.esp

Known remaining issues:
  • [upgrade] Conversion from the smbpasswd/TDB passwords to ntPwdHash / lmPwdHash is broken. Couldn't find out why.
  • [ldb_map] Conversion of attribute names in DN's is still a bit dodgy
  • [ldb_map] mapped objectClass names may be mentioned multiple times in returned records
  • [ldb_map] add/modify support not tested very well with LDAP yet (only LDB+TDB)
  • [ldb_map] group membership is not yet mapped (only primaryGroupID / sambaPrimaryGroupSID)

comments.

The alternative approach

Looking more closely at one of the other approaches I could've taken for migrating from Samba3. We will probably end up supporting this method later on anyway, once we expand the vampire code. I do think the path I've taken for SoC is the right one though - it allows the upgrade with the least hassle and user input.

This approach could be splitted up into the various parts:

  • samba3 pidl backend
  • Samba4 vampire + server side samsync support in Samba3
  • unixinfo (unixinfo) (exposes idmap) - in Samba4 (client side) - in Samba3 (server side)
  • winsrepl (thru seperate pipe?)
  • enum/add shares (srvsvc)
  • enum/add registry (winreg)
  • enum/add printers (spoolss(?))
  • convert smb.conf (using Jerry's registry hack)

Advantages of this approach:

  • Can be used to 'clone' Samba4 boxes as well
  • Relatively few extra loc necessary (a lot shared with vampire)
  • 'Clean' code - simply enumerates and adds

Disadvantages:

  • Needs running Samba3 daemon for upgrade (!)
  • Needs (complex?) Samba3 code changes
  • Needs latest version of Samba3

comments.

LDIF output working

The sam.ldif and winbind.ldif generation code now works as well. I'll be working on generating the new smb.conf next.

One of the things that worries me is the fact that I can generate so few of the Samba4 'user' and 'group' attributes. Probably need to borrow some code from the current provisioning script to fix that up.

I'm wondering whether I should do some of the upgrade using EJS, so it can be integrated into SWAT easily...

comments.

ldb_map somewhat working

ldb_map and the samba3sam ldb module that is based on it work nicely now and appear to be memory-leak free. I have so far been testing with a TDB backend loaded with an LDIF from Samba3. Tomorrow will be the big day, testing against an OpenLDAP server.

Since the LDAP schema for Samba3 contains information that the Samba4 SAM database doesn't store and vice versa, it will be interesting to find out whether the current mapping will be sufficient. If it is not, I might have to fall back to storing the Samba4 attributes I can't store in the Samba3 LDAP database in a seperate database instead.

Another thing I did today is clean up the parameter list in Samba4 - I removed all unused parameters. This should hopefully make it easier to do the Samba3->Samba4 smb.conf conversion code.

TODO is down to:

  • smb.conf conversion
  • Finish sam.ldif / winbind.ldif from the TDB files
  • Test samba3sam module against OpenLDAP

comments.

First LDIF working

Today was a productive day. I just got the LDIF output for samba3dump working for wins.ldif and the various registry hives. TODO is down to:

  • sam.ldif
  • winbind.ldif
  • smb.conf
  • test and finish ldb_map and ldb_samba3

I hope to get the first three done tomorrow.

comments.

Starting on LDIF output

Next on my list is the generation of the LDIF's for the one-way upgrade path. These will be generated by internally building ldb_ldif structs and then dumping them using the LDB helper functions. This would easily allow us to later on directly write to LDB database.

I'll need to generate the following LDIF files:

  • hklm / hkcu / etc.ldif: from samba3_regdb
  • wins.ldif: from samba3_winsdb
  • sam.ldif: from samba3_secrets, samba3_policy, samba3_privileges, samba3_samdb
  • winbind.ldif: from samba3_winbind

I'll also make a start on the smb.conf output.

comments.

ldb_map done

Finally got the ldb_map code done. This is a simple skeleton module that takes a mapping table of LDAP attributes and objectclasses and then maps between two different schema's. Also got the initial part of ldb_samba3 compiling, which is the first module that actually uses ldb_map.

Yesterday evening, I also added support for reading smb.conf files and share_info.tdb files to the lib/samba3/ directory. Looks like my TODO-list is now down to:

  • testing (writing a couple of smbtorture tests, preferably)
  • code for generating new smb.conf and for generating LDIF from the various TDB's

comments.