The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor.
The FOSS world uses a wide variety of different build tools; given a git repository or tarball, it can be hard to figure out how to build and install a piece of software.
Humans will generally know what build tool a project is using when they check out a project from git, or they can read the README. And even then, the answer may not always be straightforward to everybody. For automation, there is no obvious place to figure out how to build or install a project.
Debian
For Debian packages, Debian maintainers generally will have determined that the appropriate tools to invoke are, and added appropriate invocations to debian/rules. This is really nice when rebuilding all of Debian - one can just invoke debian/rules - a consistent interface - and it will in turn invoke the right tools to build the package, meeting a long list of requirements.
With newer versions of debhelper and most common build systems, debhelper can figure a lot of this out automatically - the maintainer just has to add the appropriate build and run time dependencies.
However, debhelper needs to be consistent in its behaviour per compat level - otherwise builds might start failing with different versions of debhelper, when the autodetection logic is changed. debhelper can also only do the right thing if all the necessary dependencies are present. debhelper also only functions in the context of a Debian package.
Ognibuild
Ognibuild is a new tool that figures out the build system in use by an upstream project, as well as the other dependencies it needs. This information can then be used to invoke said build system, or to e.g. add missing build dependencies to a Debian package.
Ognibuild uses a variety of techniques to work out what the dependencies for an upstream package are:
- Extracting dependencies and other requirements declared in build system metadata (e.g. setup.py)
- Attempting builds and parsing build logs for missing dependencies (repeating until the build succeeds), calling out to buildlog-consultant
Once it is determined which dependencies are missing, they can be resolved in a variety of ways. Apt can be invoked to install missing dependencies on Debian systems (optionally in a chroot) or ecosystem-specific tools can be used to do so (e.g. pypi or cpan). Instead of installing packages, the tool can also simply inform the user about the missing packages and commands to install them, or update a Debian package appropriately (this is what deb-fix-build does).
The target audience of ognibuild are people who need to (possibly from automation) build a variety of projects from different ecosystems or users who are looking to just install a project from source. Developers who are just hacking on e.g. a Python project are better off directly invoking the ecosystem-native tools rather than a wrapper like ognibuild.
Supported ecosystems
(Partially) supported ecosystems currently include:
- Combinations of make and autoconf, automake or CMake
- Python, including fetching packages from pypi
- Perl, including fetching packages from cpan
- Haskell, including fetching from hackage
- Ninja/Meson
- Maven
- Rust, including fetching packages from crates.io
- PHP Pear
- R, including fetching packages from CRAN and Bioconductor
For a full list, see the README.
Usage
Ognibuild provides a couple of top-level subcommands that will seem familiar to anybody who has used a couple of other build systems:
- ogni clean - remove build artifacts
- ogni dist - create a dist tarball
- ogni build - build the project in the current directory
- ogni test - run the test suite
- ogni install - install the project somewhere
- ogni info - display project information including discovered build system and dependencies
- ogni exec - run an arbitrary command but attempt to resolve issues like missing dependencies
These tools all take a couple of common options:
—resolve=apt|auto|native
Specifies how to resolve any missing dependencies:
- apt: install the appropriate dependency using apt
- native: install dependencies using native tools like pip or cpan
- auto: invoke either apt or native package install, depending on whether the current user is allowed to invoke apt
—schroot=name
Run inside of a schroot.
—explain
do not make any changes but tell the user which native on apt packages they could install.
There are also subcommand-specific options, e.g. to install to a specific directory on restrict which tests are run.
Examples
Creating a dist tarball
1 2 3 4 5 6 7 8 9 | % git clone https://github.com/dulwich/dulwich % cd dulwich % ogni --schroot=unstable-amd64-sbuild dist … Writing dulwich-0.20.21/setup.cfg creating dist Creating tar archive removing 'dulwich-0.20.21' (and everything under it) Found new tarball dulwich-0.20.21.tar.gz in /var/run/schroot/mount/unstable-amd64-sbuild-974d32d7-6f10-4e77-8622-b6a091857e85/build/tmpucazj7j7/package/dist. |
Installing ldb from source, resolving dependencies using apt
1 2 3 4 5 6 7 8 9 | % wget https://download.samba.org/pub/ldb/ldb-2.3.0.tar.gz % tar xvfz ldb-2.3.0.tar.gz % cd ldb-2.3.0 % ogni install --prefix=/tmp/ldb … + install /tmp/ldb/include/ldb.h (from include/ldb.h) … Waf: Leaving directory `/tmp/ldb-2.3.0/bin/default' 'install' finished successfully (11.395s) |
Running all tests from XML::LibXML::LazyBuilder
1 2 3 4 5 6 | % wget ``https://cpan.metacpan.org/authors/id/T/TO/TORU/XML-LibXML-LazyBuilder-0.08.tar.gz`_ <https://cpan.metacpan.org/authors/id/T/TO/TORU/XML-LibXML-LazyBuilder-0.08.tar.gz>`_ % tar xvfz XML-LibXML-LazyBuilder-0.08.tar.gz Cd XML-LibXML-LazyBuilder-0.08 % ogni test … |
Current Status
ognibuild is still in its early stages, but works well enough that it can detect and invoke the build system for most of the upstream projects packaged in Debian. If there are buildsystems that it currently lacks support for or other issues, then I’d welcome any bug reports.