Writing good mutants

A mutant is a claim: "if this line were wrong, a test would fail". Write one for each guard whose silent removal you would not notice in review. muhar is language-agnostic: it edits text and runs a command, so anything a suite can check can be mutated.

Choosing guards

Mutate the lines that refuse things. Good candidates:

• Boundary comparisons: <= against <, > against >=.
• Early returns of an error: return ErrOverLimit against return nil.
• Authorization and ownership checks: a signatory, a require, an if !allowed.
• Configuration that turns a protection on: "enforce": true, set -e, permissions: read.
• Validation of input at a trust boundary.

Skip lines whose failure would be loud anyway (a typo that stops the build), logging, and formatting. A spec of ten guards you care about is worth more than a hundred generated edits you will not read.

Name the bug, not the edit

The name is what a reader sees in the report and in CI logs. Describe the behaviour the mutant simulates:

A NOT_CAUGHT line should read as the bug report for the missing test.

The control

The control proves the harness can see a fault at all. Make it the bluntest edit available in the same file or package the mutations target: disable the main refusal, return success unconditionally, delete the check. It must fail the suite on every run, and it should fail it for the same reason a real regression would — through the tests, not through a build error in an unrelated file.

Keep the control in the same tree and build unit as the mutations. A control in another module proves that module is wired up, not this one.

Old text that survives edits

old is matched exactly and replaced once (spec reference). Three habits keep it stable:

• Include enough context to be unique — usually the whole condition, not one operator.
• Leave out neighbouring lines that change often: comments, log messages, version strings.
• When a refactor moves the guard, muhar reports COULD_NOT_MUTATE, Old text not found. Treat that as a red result and update the spec in the same change. It is the harness telling you the guard moved, not noise.

In JSON, write tabs as \t and newlines as \n. Prefer a single-line old where you can: it is easier to read in review and does not depend on line endings.

Go

{
  "name": "an over-limit withdrawal is allowed",
  "file": "limits.go",
  "old": "\t\treturn ErrOverLimit",
  "new": "\t\treturn nil"
}

Run with "test": "go test ./... -count=1 -timeout 60s". -count=1 disables Go's test result cache so every mutant really runs; -timeout bounds a mutant that makes a loop spin (timeouts). A mutant that no longer compiles fails go test and is reported CAUGHT; see the FAQ.

JSON configuration

A configuration file is a guard when a test or a validator reads it. Mutate the value that turns the protection on:

{
  "name": "the rate limit is switched off",
  "file": "config/limits.json",
  "old": "\"enforce\": true",
  "new": "\"enforce\": false"
}

This is only caught if some test loads config/limits.json and asserts on behaviour that depends on it. A NOT_CAUGHT here often means the tests use a fixture instead of the shipped file.

Shell scripts

Shell guards are flags and early exits. A test for a script is any command that runs it against a case it must refuse:

{
  "root": ".",
  "test": "sh tests/run.sh",
  "control": {
    "name": "CONTROL: the deploy script never refuses",
    "file": "scripts/deploy.sh",
    "old": "  echo \"refusing: working tree is dirty\" >&2\n  exit 1",
    "new": "  echo \"refusing: working tree is dirty\" >&2"
  },
  "mutations": [
    {
      "name": "a failing step no longer stops the script",
      "file": "scripts/deploy.sh",
      "old": "set -eu",
      "new": "set -u"
    }
  ]
}

tests/run.sh might create a temporary repository with an uncommitted file, run scripts/deploy.sh in it, and exit non-zero unless the script exited non-zero. The set -eu mutant is caught only if some test makes an intermediate step fail and checks the script stops.

YAML workflows

CI workflows carry permissions and conditions that are easy to loosen in review. They are guards when a linter or policy check reads them:

{
  "name": "the workflow token can write to the repository",
  "file": ".github/workflows/build.yml",
  "old": "  contents: read",
  "new": "  contents: write"
}

With "test": "./scripts/check-workflows.sh", where that script fails on any workflow granting write access it does not allow. Without such a check, this mutant is NOT_CAUGHT by design — and that is the finding.

One spec per suite

A spec has one test command. If different guards are covered by different suites — Go tests, shell tests, a workflow linter — write one spec per suite and run each as its own step or CI job. Each one then proves its own baseline and control, and a broken shell suite cannot be hidden by a healthy Go run.

Reviewing a spec

Before merging a spec change, check:

• The control is the bluntest available edit in the same build unit as the mutations.
• Every old occurs in its file, and the first occurrence is the one intended.
• Every name describes a bug a reader would recognise.
• The test command has a timeout and does not reuse cached results.
• The spec runs green locally: exit 0, 0 NOT CAUGHT, 0 could-not-mutate.

Validate the shape against the JSON Schema to catch misspelt keys, which muhar's decoder ignores.