# How a run works

This page follows one invocation from start to exit. It is written against the source in [tools/desk/cmd/muhar](https://github.com/medici-finance/assay/tree/main/tools/desk/cmd/muhar): `main.go` (flags, spec, environment, exit), `harness.go` (the run, the verdicts, the report) and `workspace.go` (worker copies).

## Start-up

1. Parse flags. A flag error exits 2 from Go's flag parser.
2. Read and decode the spec. Require a non-empty `test` and a control with a `name` and `old`. Any failure exits 1.
3. With `-shard i/n`: keep only the mutations at index ≡ i (mod n) and print the selection to stderr.
4. Resolve `-j`: `0` becomes the CPU count capped at 2; negative exits 1.
5. Create the per-run Go build cache, a fresh temp directory named `assay-mutant-gocache-*`.
6. Install a handler for `SIGINT` and `SIGTERM` ([Signals](#signals)).
7. With `-j` above 1: name the tree to copy (`root`, or the current directory) and print the `-j` line to stderr.

## Baseline and control

The suite runs once on pristine source. A non-zero exit discards the run. Then the control is planted exactly like a mutation (below) and the suite runs again. Anything other than `CAUGHT` discards the run. A discarded run prints its reason, removes the build cache and exits 2 without touching any other mutation.

With at least two effective workers, both steps run inside the first worker's copy, not in the original tree. This proves the copy builds and tests the same as the original before any verdict depends on it. The worker count is capped at the number of mutations in the spec (or shard); with one, execution falls back to running in place like `-j 1`, including the baseline and control.

## Planting one mutation

For the control and for each mutation, in a given tree:

1. Resolve `file`: absolute as is, relative joined to the tree's root.
2. Read the file. Failure: `COULD_NOT_MUTATE`.
3. Compute the edit in memory. Refuse an empty `old`, `old` equal to `new`, `old` not found, or a result identical to the input: `COULD_NOT_MUTATE`.
4. Write the edited content with the file's existing permission bits.
5. Read the file back. If the bytes differ from the intended content, or equal the original, `COULD_NOT_MUTATE`.
6. Run the suite. Non-zero: `CAUGHT`. Zero: `NOT_CAUGHT`.
7. Write the original bytes back. This happens once step 4 has succeeded, whatever the verdict.

Step 5 is the [proven-landed guarantee](concepts.md#proven-landed). Steps 2, 3 and 5 are why a `NOT_CAUGHT` always means the fault was in place.

Figure: The order of one run. A failure in the first two steps discards the run with exit 2; a mutation that cannot be planted is recorded and the run continues.

1. Baseline: run the suite on pristine source. It must pass, or the run is discarded (exit 2).
2. Positive control: plant, prove on disk, run the suite, restore. It must be CAUGHT, or the run is discarded (exit 2).
3. Each mutation, in spec order: plant, prove on disk, run the suite, restore. A mutation that cannot be planted or proven is COULD_NOT_MUTATE and the suite is not run.
4. Report: per-mutation verdicts in spec order, then the Totals line. Exit 0.

## The suite process

The suite is `sh -c '<test>'`, started with its working directory set to the tree's root and its environment set to muhar's own with `GOCACHE` replaced. Its stdout and stderr both go to muhar's stderr. muhar waits for it to exit; only the exit status is used.

## Worker copies

With `-j k` above 1, muhar caps the worker count at the number of mutations. If at least two workers remain, it makes one copy of the tree per worker before the baseline, each in a new temp directory named `muhar-<i>-*`. With one mutation, it makes no copies and runs in place, even with `-j 2` or `-j 0`.

Figure: -j 2: the original tree is copied once per worker. The baseline and control run in copy 0; mutations are dealt to whichever worker is free; results are reported in spec order.

Original tree (root): read only during the run, except for absolute file paths.
Copy 0 (worker 0): baseline, then control, then mutations as dealt.
Copy 1 (worker 1): mutations as dealt.
Both workers share the one per-run GOCACHE.

The copy is a full deep copy: regular files with their permission bits, directories, and symbolic links recreated as links with the same target text. Sockets, devices and other special files are skipped. Everything under `root` is copied, including `.git`, `node_modules` and build output, so point `root` at the smallest tree the suite needs.

After the preflight, mutation indices are fed to the workers one at a time; each worker plants, tests and restores in its own copy. Results are stored by spec index, so the report does not depend on which worker finished first. When the run ends normally, every copy is removed.

### go.mod replace directives

A `replace example.com/x => ../x` in `go.mod` is resolved relative to that `go.mod`'s directory. In a copy under the temp directory, a relative target that points outside the copied tree resolves to nothing, every build in the copy fails, and the baseline would discard the run.

So, for every `go.mod` in each copy (skipping `testdata` and directories whose names start with `.` or `_`, which the go command ignores), muhar reads the replace directives with `go mod edit -json` and rewrites each one that is a filesystem path, relative, and outside the source tree to the absolute path of its target in the original tree. It then re-reads the directives and fails the copy if any rewrite did not land. Relative targets inside the tree are left alone — they were copied, and resolving them inside the copy keeps mutations to them isolated. Versioned replacements and absolute paths already resolve from anywhere.

This uses the `go` command, which must be on `PATH` when any copied `go.mod` contains the word `replace`. A failure to rewrite is a workspace failure, reported as `HARNESS BROKEN`.

## The per-run Go build cache

Every mutant is a source tree the Go build cache has never seen, so each suite run adds entries that will never be used again. In a long-lived checkout or CI cache that directory grows without bound; Go trims unused entries only after five days, and that horizon is not configurable.

muhar therefore gives the suite a fresh `GOCACHE` for the whole run, shared by every suite run and every worker in it, and removes it at the end. Mutants of one package share most of their build, so one cache per run keeps that speed-up within the run. `GOMODCACHE` is left alone: downloaded modules are immutable and safe to share.

The cache is removed on a normal exit, on a broken run, and on `SIGINT` or `SIGTERM`.

## Signals and cleanup

On `SIGINT` or `SIGTERM` muhar removes the per-run build cache and exits 130 at once. It does not wait for the suite or run any other clean-up:

- In a sequential run (including the one-mutation fallback with `-j` above 1), the file being mutated at that moment is left mutated on disk. Check `git status` and restore it with `git checkout -- <file>` (or `git restore <file>`) before you commit.
- When two or more effective workers are used, the worker copies are left in the temp directory as `muhar-<i>-*`. Remove them by hand.
- After `SIGKILL`, the build cache is left too, as `assay-mutant-gocache-*`.

This is a known gap, tracked as [assay#2641](https://github.com/medici-finance/assay/issues/2641). Until it is fixed, run muhar on a clean working tree, so an interrupted run is obvious in `git status`.

## Timeouts

muhar has no timeout of its own. A mutation that makes the suite hang — a loop condition that never becomes false — hangs muhar. Put the limit in the `test` command:

- `go test ./... -timeout 60s` fails the test binary when the limit is hit, so the mutation is reported `CAUGHT`.
- For any other suite, wrap it: `timeout 120 make test`. GNU `timeout` exits 124 when the limit is hit, which is non-zero, so `CAUGHT`.

Set the limit well above the baseline's run time, or a slow machine turns the baseline red and discards the run. A per-mutation timeout is planned ([roadmap](roadmap.md)).

## What a run costs

Each invocation runs the suite `2 + M` times: baseline, control, and one per mutation. With `-shard i/n` it is `2 + ⌈M/n⌉` or fewer per shard; every shard pays the 2. With `-j k` the `M` mutation runs proceed `k` at a time, after the `k` copies are made and the preflight has finished.

The whole suite runs for every mutation. muhar does not yet select only the tests that reach the mutated code, and every Go mutant rebuilds the mutated package and whatever imports it. Work to reduce both is in progress: build-cost reductions and measured `-j` sizing ([assay#2635](https://github.com/medici-finance/assay/issues/2635)), and coverage-guided test selection ([assay#2636](https://github.com/medici-finance/assay/issues/2636)). Neither is released; see the [roadmap](roadmap.md).

---

Source of this page: https://muhar.assay.guide/internals