CI integration

A muhar step in CI has three parts: install a pinned muhar, run the spec, and fail unless the exit status is 0 and the Totals: line shows zero NOT CAUGHT and zero could-not-mutate. Everything on this page builds on the gate rule.

A single job

GitHub Actions, one spec, one runner:

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
      - name: Install muhar
        env:
          BASE: https://github.com/medici-finance/assay/releases/download/v1.0.39
        run: |
          set -euo pipefail
          curl -fsSLO "$BASE/desk-tools-linux-amd64.tar.gz"
          curl -fsSLO "$BASE/checksums.txt"
          grep " desk-tools-linux-amd64.tar.gz\$" checksums.txt | sha256sum -c -
          tar -xzf desk-tools-linux-amd64.tar.gz ./muhar
          install -m 0755 muhar "$RUNNER_TEMP/muhar"
      - name: Mutation gate
        shell: bash
        run: |
          set -uo pipefail
          status=0
          report=$("$RUNNER_TEMP/muhar" -spec mutations.json) || status=$?
          printf '%s\n' "$report"
          if [ "$status" -ne 0 ]; then
            echo "::error::muhar exited $status: harness broken or usage error, no verdict"
            exit 1
          fi
          if ! grep -qE '^Totals: [0-9]+ caught, 0 NOT CAUGHT, 0 could-not-mutate\.$' <<<"$report"; then
            echo "::error::a guard has no test, or a mutation did not land"
            exit 1
          fi

Notes:

• The step uses set -uo pipefail without -e, so a non-zero muhar exit is captured in status and reported, instead of ending the step before the message.
• The suite's output goes to stderr and appears in the log above the report. The report itself is printed once, by printf.
• actions/setup-go is needed only because this example's suite is go test. muhar itself needs no Go toolchain, except for the replace rewrite with -j above 1.
• Pin the release in BASE. With latest, a new release can change the gate without a commit.

Splitting across matrix legs

A large spec can take longer than one job should. -shard i/n runs a disjoint share of the mutations; n legs with i from 0 to n-1 cover every mutation exactly once (CLI reference). Every leg proves the baseline and control on its own, so a broken harness fails all of them.

jobs:
  mutation:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: ["0/3", "1/3", "2/3"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
      - name: Install muhar
        run: ./ci/install-muhar.sh
      - name: Mutation gate, shard ${{ matrix.shard }}
        shell: bash
        env:
          MUHAR_SHARD: ${{ matrix.shard }}
        run: |
          set -uo pipefail
          status=0
          report=$(muhar -j 0 -shard "$MUHAR_SHARD" -spec mutations.json) || status=$?
          printf '%s\n' "$report"
          mkdir -p "$RUNNER_TEMP/muhar-reports"
          printf '%s\n' "$report" > "$RUNNER_TEMP/muhar-reports/shard-${MUHAR_SHARD%/*}.txt"
          if [ "$status" -ne 0 ]; then
            echo "::error::shard $MUHAR_SHARD: muhar exited $status, no verdict"
            exit 1
          fi
          grep -qE '^Totals: [0-9]+ caught, 0 NOT CAUGHT, 0 could-not-mutate\.$' <<<"$report" || {
            echo "::error::shard $MUHAR_SHARD: a guard has no test, or a mutation did not land"
            exit 1
          }
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: muhar-shard-${{ strategy.job-index }}
          path: ${{ runner.temp }}/muhar-reports/

./ci/install-muhar.sh stands for the install step from the single-job example; it must put muhar on PATH.

• Pass the shard through an environment variable rather than interpolating ${{ matrix.shard }} into the script. It keeps the expression out of the shell text.
• fail-fast: false lets every leg finish, so one run shows every missing test, not the first shard's.
• The matrix list must name every i/n for one n. A missing leg silently drops its mutations from the gate. Generate the list, or keep it next to a comment stating n.
• Each leg's Totals: line covers that leg only. The gate is the conjunction: every leg green.
• Every leg pays for the baseline and the control. Shard when the mutations dominate the run time, not for a spec of three.

Sizing -j

-j 0 uses one worker per CPU, capped at 2, which bounds peak memory on large runners (why). On a runner with headroom, an explicit -j 4 may be faster; measure it, because each worker holds a full copy of root and its own suite process. Combine -j with -shard: shard across machines, -j within one.

Point root at the smallest tree the suite needs. Every worker copies everything under it, including .git and node_modules.

What to keep

Keep the report — stdout — as a build artifact, as in the matrix example. The suite's output on stderr is in the job log. A COULD_NOT_MUTATE line names the mutation and why it did not land, so the report alone is enough to fix the spec.

Other CI systems

The gate is plain bash, so it carries over unchanged. In GitLab CI:

mutation:
  image: golang:1.25
  parallel:
    matrix:
      - MUHAR_SHARD: ["0/3", "1/3", "2/3"]
  script:
    - ./ci/install-muhar.sh
    - ./ci/muhar-gate.sh

where ci/muhar-gate.sh holds the bash gate, reading MUHAR_SHARD. Any system that runs a script and fails on a non-zero exit can run it.

How Assay uses it

Assay's release workflow runs muhar on its own merge tool: three matrix legs, -shard 0/3, 1/3 and 2/3, each go run ./cmd/muhar -j 0 -shard "$MUHAR_SHARD" -spec cmd/deskmerge/mutations.json from the tools/desk module, gated on the same Totals: pattern. A second leg runs six smaller specs one after another with -j 0. Reports are written to the runner's temp directory.

Coming

A reusable CI action that installs muhar, runs a sharded sweep and applies this gate is planned (assay#2642), as is a diff-scoped mode that runs only the mutations touching changed files (assay#2640). Neither exists yet; see the roadmap.