Posts tagged 'conferences'

subunit usage in Samba

Both Samba 3 and Samba 4 are now using the "subunit" protocol inside their testsuite (aka "make test"). subunit is a streaming protocol used to report test results that is aimed at being simple to generate and parse as well as being human readable.

A very simple subunit stream might look like this:

test: samba4.tests.util.strlist.check_list_make
creating list...
list created!
success: samba4.tests.util.strlist.check_list_make
test: samba4.tests.util.strlist.check_list_make_shell
creating list...
xfail: samba4.tests.util.strlist.check_list_make_shell [
returned NT_STATUS_NOT_IMPLEMENTED
]

For those that are familiar with the TAP protocol used by Perl, it is similar to that, although it has a couple of features that TAP does not have. For example, it can report timestamps (useful for determining test duration) and has more flexible progress reporting.

Subunit is particularly useful for projects that use multiple programming languages as it allows a single tool to be used for test visualization or analysis rather than one per language. All that's required per-language is a test runner that can spit out subunit streams.

selftest.pl, the main engine behind Samba's test suite, has been using subunit internally since its creation a couple of years ago. Most other test tools we use can also report subunit, in particular our Python tests, blackbox tests, Perl tests (using tap2subunit) and smbtorture.

"make test" never displays raw subunit results, it always formats them using our format-subunit script. Samba 4's "make test" stores the raw subunit output in st/subunit.

I'm attending SNIA SDC at the moment and a couple of people here have asked me about the tools I use to display and analyse test results. They are:

subunit

The subunit project contains a bunch of convenience tools for working with subunit. Other than libraries for parsing/generating subunit for several languages it contains tools for manipulating and analysing subunit streams, including:

  • subunit-ls: List all tests in a subunit stream, optionally including their run times (I used this for the test duration summary I sent to the Samba mailing list earlier)
  • tap2subunit: convert a TAP stream to a Subunit stream
  • subunit-stats: Print statistics for a subunit stream (how many successful tests, failed tests, skipped tests, etc)
  • subunit-filter: E.g. remove test result or output from a stream
  • subunit-diff: Compare two subunit streams and see what tests have started failing or are no longer failing
  • subunit2pyunit: Format a subunit stream using Python's standard unit test test result formatter

We're including the subunit tree in the Samba git tree at lib/subunit.

tribunal

Tribunal is a GTK+ viewer for subunit streams. It allows for easy browsing of test results. Tribunal is still a bit rough around the edges, although it should already be useful.

Example usage:

$ make test
$ tribunal-subunit st/subunit

testrepository

Test Repository provides a database of test results which fits into developers work flow and keeps track of useful information like what tests are failing, or which failures have the same backtrace.

In particular Test Repository can re-run only the tests that failed in the previous test run:

$ testr init
# Run the full testsuite (1 hour goes by)
$ testr run
# Run those tests from the testsuite that failed in the previous run
# (this would be a lot shorter usually, depending on how many tests were
# failing)
$ testr run --failing

testrepository is also still in its early days, but can potentially be very useful, e.g. when comparing old test runs on the buildfarm.

comments.

SambaXP and other travel

It's been a busy two weeks. Wilco and I drove up to Göttingen on Sunday two weeks ago to spend some days hacking and meeting up with the other developers before the start of SambaXP. It was really nice to see everybody again after more than 7 months.

SambaXP was a bit different this year. There were three tracks during the second part of the conference this year, one more than previously and of course, there were several engineers from Microsoft attending this time! Some of the interesting talks this year included Julien's update on OpenChange, Tridge's talk on PFIF, the talk from the likewise folks and of course the talk from Microsofts' Wolfgang Grieskamp on SMB2. We also had some other informal discussions with the Microsoft folks about specific topics - very useful!

There are some photos up on the SambaXP homepage. And just to be ahead of the comments: yes, I know I need a haircut.

I did some initial work on several bits and pieces of code that I hope to expand over the next few months. Volker has started working on ncacn_ip_tcp support and I have been working on making the Samba 3 DCE/RPC library compatible with Samba 4. This should allow OpenChange to use Samba 3 in the future.

Guenther, Wilco and I made some initial progress on the policy library, allowing client-side manipulation of (group) policies in Samba. I worked with Simo on trying to get rid of an evil hack in Samba4's event subsystem.

David Holder blogged about some of the IPv6 development that we did during the conference: http://www.ipv6consultancy.com/ipv6blog/?p=34

And lots of other things I can't remember at the moment...

After the conference Andrew, Wilco and I drove back to the Netherlands and I played tour guide for a bit showing Andrew around the country during the afternoon and hacking Samba together in the morning. Later this week we took the train to Brussels, Eurostar to London and visited Sam's company in the UK Midlands for a couple of days.

And in the midst of all this, it seems Ubuntu Hardy was released. Congratulations to all those involved!

Currently Playing: Brandi Carlile - Turpentine

comments.

Podcast with the Samba team

Last week was the annual Samba get-together in the US. Jeremy and Leslie from Google organized a somewhat more informal event than the SNIA CIFS interop lab that we usually attend.

Several Samba team members, including myself, took part in a podcast about our involvement in the Summer of Code afterwards.

comments.