The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor.
In my last blogpost, I introduced the buildlog consultant - a tool that can identify many reasons why a Debian build failed.
For example, here’s a fragment of a build log where the Build-Depends lack python3-setuptools:
849 850 851 852 853 854 855 856 857 858 | dpkg-buildpackage: info: host architecture amd64 fakeroot debian/rules clean dh clean --with python3,sphinxdoc --buildsystem=pybuild dh_auto_clean -O--buildsystem=pybuild I: pybuild base:232: python3.9 setup.py clean Traceback (most recent call last): File "/<<PKGBUILDDIR>>/setup.py", line 2, in <module> from setuptools import setup ModuleNotFoundError: No module named 'setuptools' E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean |
The buildlog consultant can identify the line in bold as being key, and interprets it:
% analyse-sbuild-log --json ~/build.log
{
"stage": "build",
"section": "Build",
"lineno": 857,
"kind": "missing-python-module",
"details": {"module": "setuptools", "python_version": 3, "minimum_version": null}
}
Automatically acting on buildlog problems
A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends.
Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to Build-Depends. In the example given above, we can use apt-file to look for the package that contains the path /usr/lib/python3/dist-packages/setuptools/__init__.py - and voila, we find python3-setuptools:
% apt-file search /usr/lib/python3/dist-packages/setuptools/__init__.py
python3-setuptools: /usr/lib/python3/dist-packages/setuptools/__init__.py
The deb-fix-build command automates these steps:
- It builds the package using sbuild; if the package successfully builds then it just exits successfully
- It tries to identify the problem by looking through the build log; if it can’t or if it’s a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code
- It tries to find a dependency that can address the problem
- It updates Build-Depends in debian/control or Depends in debian/tests/control
- Go to step 1
This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again.
For example, when I ran deb-fix-build while packaging saneyaml, the output looks something like this:
% deb-fix-build
Using output directory /tmp/tmpyz0nkgqq
Using sbuild chroot unstable-amd64-sbuild
Using fixers: …
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4')
Using apt-file to search apt contents
Adding build dependency: python3-setuptools-scm (>= 4)
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None)
Adding build dependency: python3-toml
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Built 0.5.2-1- changes files at [‘saneyaml_0.5.2-1_amd64.changes’].
And in our Git repository, we see these changes as well:
% git log -p
commit 5a1715f4c7273b042818fc75702f2284034c7277 (HEAD -> master)
Author: Jelmer Vernooij <jelmer@jelmer.uk>
Date: Sun Apr 4 02:35:56 2021 +0100
Add missing build dependency on python3-toml.
diff --git a/debian/control b/debian/control
index 5b854dc..3b27b73 100644
--- a/debian/control
+++ b/debian/control
@@ -1,6 +1,6 @@
Rules-Requires-Root: no
Standards-Version: 4.5.1
-Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)
+Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4), python3-toml
Testsuite: autopkgtest-pkg-python
Source: python-saneyaml
Priority: optional
commit f03047da80fcd8468ee231fbc4cf8488d7a0acd1
Author: Jelmer Vernooij <jelmer@jelmer.uk>
Date: Sun Apr 4 02:35:34 2021 +0100
Add missing build dependency on python3-setuptools-scm (>= 4).
diff --git a/debian/control b/debian/control
index a476cc2..5b854dc 100644
--- a/debian/control
+++ b/debian/control
@@ -1,6 +1,6 @@
Rules-Requires-Root: no
Standards-Version: 4.5.1
-Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel
+Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)
Testsuite: autopkgtest-pkg-python
Source: python-saneyaml
Priority: optional
Using deb-fix-build
You can run deb-fix-build by installing the ognibuild package from unstable. The only requirements for using it are that:
- The package is maintained in Git
- A sbuild schroot is available for use
Caveats
deb-fix-build is fairly easy to understand, and if it doesn’t work then you’re no worse off than you were without it - you’ll have to add your own Build-Depends.
That said, there are a couple of things to keep in mind:
- At the moment, it doesn’t distinguish between general, Arch or Indep Build-Depends.
- It can only add dependencies for things that are actually in the archive
- Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn’t always get it right