Jelmer Vernooij: Inquest, a test result repository in Rust

testrepository

For a long time I’ve used Robert Collins’ testrepository (testr) to
run tests in many of the projects I work on. It’s a small, focused tool
built around a simple idea: decouple the running of tests from the
recording and querying of their results.

The way it works is straightforward. A test runner emits a subunit
stream — a compact binary protocol for test results — and testrepository
stores those streams in a per-project .testrepository/ directory.
Once results are in the repository, you can ask questions like “which
tests failed in the last run?”, “re-run only the failures”, “what are the
slowest tests?”, or “what changed between this run and the previous one?”.

The killer feature, for me, has always been the failing-test loop. When a
big test suite breaks, you don’t want to re-run the whole thing after every
fix — you want to iterate on just the failures, and only re-run the full
suite once they’re all green. testrepository made that workflow ergonomic
long before most language-specific test runners had anything comparable,
and many of them still don’t have a good answer for it.

testrepository has served me well for over a decade, but it has been
largely unmaintained for a while, and I had some ideas of improvements that I
wanted to try out. So I wrote a Rust port, which has since grown a number of
features of its own.

Inquest

Inquest is a Rust port of testrepository that has since grown a
number of features of its own. The binary is called inq.

Goals

The goals are deliberately modest:

• a single static binary, no Python runtime required
• no need to write a dedicated config file for most projects
• compatible enough with testrepository’s workflow that I can switch projects
  over without retraining my fingers
• a richer on-disk format that captures more about each run (git commit,
  command line, duration, exit code, concurrency)
• good support for the languages I actually use day-to-day: Rust, Python,
  Go, and Node.js
• mostly Do What I Mean (DWIM), e.g. getting me to know as quickly
  as possible what tests are failing and why, and being clever
  about doing this

Inquest reads and writes subunit v2 streams, so anything that can produce
subunit (directly or via one of the many converters) can feed into it.

Quick start

Inquest can usually figure out how to run your tests on its own. In a Rust,
Python, Go or Node.js project:

 $ cd my-project
 $ inq

Or if the auto-detection doesn’t work, you can ask it to generate a config file and
then run the tests:

 $ inq auto
 $ inq run

inq auto writes an inquest.toml describing how to invoke the test
runner; inq run runs the tests, captures the subunit stream, and stores
the results in a .inquest/ directory.

For a Rust project the generated config looks like:

 test_command = "cargo subunit $IDOPTION"
 test_id_option = "--test $IDFILE"
 test_list_option = "--list"

After the first run, the usual queries work:

 $ inq stats             # repository-wide statistics
 $ inq last              # results of the most recent run
 $ inq failing           # only the failing tests
 $ inq slowest           # the slowest tests in the last run
 $ inq run --failing     # re-run only what failed last time

The last one is the workflow I use most often: run the full suite once, fix
the obvious failures, then iterate on inq run --failing until the list
is empty.

A few things that aren’t in testrepository

Some of the features that have grown in inquest beyond the original
testrepository functionality:

• Timeouts. --test-timeout, --max-duration, and
  --no-output-timeout will kill a test process that is hanging or has
  stopped producing output. --test-timeout auto derives a per-test
  timeout from the historical duration of that test, which is handy for
  catching tests that hang.

  Once the test runner is killed, the test is marked as failed and the next
  test is started, so a broken test doesn’t hold up the whole suite.

• Ordering --order can be used to run tests in a specific order, e.g.
  to run the slowest tests first, to run the tests that failed most recently
  first, or to run the widest variety of tests first to maximize the chance of
  finding a failure early on.

• Live progress. inq running tails the in-progress subunit stream
  on disk and reports observed/expected test counts, percent complete,
  elapsed wall-clock time, and an ETA derived from each test’s historical
  duration. Useful when a CI run is taking longer than you’d like.

• Flakiness ranking. inq flaky ranks tests by pass↔fail transitions
  in consecutive runs in which the test was recorded, so chronically broken
  tests rank low and genuinely flapping tests rank high.

• Comparing runs. inq diff <A> <B> shows what changed between two
  test runs — newly failing, newly passing, and tests that flipped state —
  which makes it easy to see whether your last change actually fixed (or
  broke) anything.

• Bisecting git history. inq bisect <TEST> drives git bisect
  to find the commit that broke a given test. It defaults the
  known-good and known-bad commits from the recorded run history (the
  most recent run where the test passed, and the most recent where it
  failed), so in the common case there is no need to remember either —
  just point it at the test name and let it work.

• Richer run metadata. inq info shows the git commit, command
  line, duration, exit code, and concurrency for a run, with a flag for
  whether the working tree was dirty when the run started. Combined with
  inq diff this makes it much easier to triangulate when a regression
  was introduced.

• Rerun a previous run verbatim. inq rerun <ID> re-runs exactly the
  tests of a previous run, in the same order, forwarding the same --
  arguments that the original run used. inq rerun -1 repeats the latest.

• Web based view. inq web serves a web-based view of the repository,
  with a dashboard of recent runs and detailed views of individual runs and tests.

Migrating from testrepository

If you already have a .testrepository/ directory full of historical
runs, inq upgrade will migrate it into the new .inquest/ format,
with a progress bar for the impatient.

The legacy .testr.conf (INI) format is still understood, so existing
projects don’t have to be converted to inquest.toml immediately —
though the TOML format is preferred for new projects.

Trying it

The source is on GitHub at jelmer/inquest. To install from source:

 $ cargo install inquest

In a project with a Rust, Python, Go or Node.js test suite:

 $ inq

Bug reports and patches are welcome.