How a run works

This page follows one invocation from start to exit. It is written against the source in tools/desk/cmd/muhar: main.go (flags, spec, environment, exit), harness.go (the run, the verdicts, the report) and workspace.go (worker copies).

Start-up

1. Parse flags. A flag error exits 2 from Go's flag parser.
2. Read and decode the spec. Require a non-empty test and a control with a name and old. Any failure exits 1.
3. With -shard i/n: keep only the mutations at index ≡ i (mod n) and print the selection to stderr.
4. Resolve -j: 0 becomes the CPU count capped at 2; negative exits 1.
5. Create the per-run Go build cache, a fresh temp directory named assay-mutant-gocache-*.
6. Install a handler for SIGINT and SIGTERM (Signals).
7. With -j above 1: name the tree to copy (root, or the current directory) and print the -j line to stderr.

Baseline and control

The suite runs once on pristine source. A non-zero exit discards the run. Then the control is planted exactly like a mutation (below) and the suite runs again. Anything other than CAUGHT discards the run. A discarded run prints its reason, removes the build cache and exits 2 without touching any other mutation.

With at least two effective workers, both steps run inside the first worker's copy, not in the original tree. This proves the copy builds and tests the same as the original before any verdict depends on it. The worker count is capped at the number of mutations in the spec (or shard); with one, execution falls back to running in place like -j 1, including the baseline and control.

Planting one mutation

For the control and for each mutation, in a given tree:

1. Resolve file: absolute as is, relative joined to the tree's root.
2. Read the file. Failure: COULD_NOT_MUTATE.
3. Compute the edit in memory. Refuse an empty old, old equal to new, old not found, or a result identical to the input: COULD_NOT_MUTATE.
4. Write the edited content with the file's existing permission bits.
5. Read the file back. If the bytes differ from the intended content, or equal the original, COULD_NOT_MUTATE.
6. Run the suite. Non-zero: CAUGHT. Zero: NOT_CAUGHT.
7. Write the original bytes back. This happens once step 4 has succeeded, whatever the verdict.

Step 5 is the proven-landed guarantee. Steps 2, 3 and 5 are why a NOT_CAUGHT always means the fault was in place.

The suite process

The suite is sh -c '<test>', started with its working directory set to the tree's root and its environment set to muhar's own with GOCACHE replaced. Its stdout and stderr both go to muhar's stderr. muhar waits for it to exit; only the exit status is used.

Worker copies

With -j k above 1, muhar caps the worker count at the number of mutations. If at least two workers remain, it makes one copy of the tree per worker before the baseline, each in a new temp directory named muhar-<i>-*. With one mutation, it makes no copies and runs in place, even with -j 2 or -j 0.

The copy is a full deep copy: regular files with their permission bits, directories, and symbolic links recreated as links with the same target text. Sockets, devices and other special files are skipped. Everything under root is copied, including .git, node_modules and build output, so point root at the smallest tree the suite needs.

After the preflight, mutation indices are fed to the workers one at a time; each worker plants, tests and restores in its own copy. Results are stored by spec index, so the report does not depend on which worker finished first. When the run ends normally, every copy is removed.

go.mod replace directives

A replace example.com/x => ../x in go.mod is resolved relative to that go.mod's directory. In a copy under the temp directory, a relative target that points outside the copied tree resolves to nothing, every build in the copy fails, and the baseline would discard the run.

So, for every go.mod in each copy (skipping testdata and directories whose names start with . or _, which the go command ignores), muhar reads the replace directives with go mod edit -json and rewrites each one that is a filesystem path, relative, and outside the source tree to the absolute path of its target in the original tree. It then re-reads the directives and fails the copy if any rewrite did not land. Relative targets inside the tree are left alone — they were copied, and resolving them inside the copy keeps mutations to them isolated. Versioned replacements and absolute paths already resolve from anywhere.

This uses the go command, which must be on PATH when any copied go.mod contains the word replace. A failure to rewrite is a workspace failure, reported as HARNESS BROKEN.

The per-run Go build cache

Every mutant is a source tree the Go build cache has never seen, so each suite run adds entries that will never be used again. In a long-lived checkout or CI cache that directory grows without bound; Go trims unused entries only after five days, and that horizon is not configurable.

muhar therefore gives the suite a fresh GOCACHE for the whole run, shared by every suite run and every worker in it, and removes it at the end. Mutants of one package share most of their build, so one cache per run keeps that speed-up within the run. GOMODCACHE is left alone: downloaded modules are immutable and safe to share.

The cache is removed on a normal exit, on a broken run, and on SIGINT or SIGTERM.

Signals and cleanup

On SIGINT or SIGTERM muhar removes the per-run build cache and exits 130 at once. It does not wait for the suite or run any other clean-up:

• In a sequential run (including the one-mutation fallback with -j above 1), the file being mutated at that moment is left mutated on disk. Check git status and restore it with git checkout -- <file> (or git restore <file>) before you commit.
• When two or more effective workers are used, the worker copies are left in the temp directory as muhar-<i>-*. Remove them by hand.
• After SIGKILL, the build cache is left too, as assay-mutant-gocache-*.

This is a known gap, tracked as assay#2641. Until it is fixed, run muhar on a clean working tree, so an interrupted run is obvious in git status.

Timeouts

muhar has no timeout of its own. A mutation that makes the suite hang — a loop condition that never becomes false — hangs muhar. Put the limit in the test command:

• go test ./... -timeout 60s fails the test binary when the limit is hit, so the mutation is reported CAUGHT.
• For any other suite, wrap it: timeout 120 make test. GNU timeout exits 124 when the limit is hit, which is non-zero, so CAUGHT.

Set the limit well above the baseline's run time, or a slow machine turns the baseline red and discards the run. A per-mutation timeout is planned (roadmap).

What a run costs

Each invocation runs the suite 2 + M times: baseline, control, and one per mutation. With -shard i/n it is 2 + ⌈M/n⌉ or fewer per shard; every shard pays the 2. With -j k the M mutation runs proceed k at a time, after the k copies are made and the preflight has finished.

The whole suite runs for every mutation. muhar does not yet select only the tests that reach the mutated code, and every Go mutant rebuilds the mutated package and whatever imports it. Work to reduce both is in progress: build-cost reductions and measured -j sizing (assay#2635), and coverage-guided test selection (assay#2636). Neither is released; see the roadmap.