Building an Eval Harness You Will Actually Run

The scariest deploy is a one-line prompt tweak. It looks harmless in a diff and can quietly degrade quality for everyone. I wanted the same safety net for prompts that I have for code.

The Problem

We had no way to know if a prompt change made things better or worse. "It seems fine on my three examples" is not a test.

Investigation

Heavy eval frameworks existed, but nobody ran them. The friction killed adoption. The winning constraint became: it must run in CI in under a minute, or it will rot.

The Solution

A small dataset of graded cases, a model-based grader for open-ended output, and a hard threshold that fails the build. That is the whole thing.

for (const testCase of dataset) {
  const output = await runPrompt(testCase.input)
  const score = await grade(output, testCase.rubric)
  results.push({ ...testCase, score })
}

const avg = mean(results.map((r) => r.score))
if (avg < THRESHOLD) process.exit(1)

The Result

We caught two real regressions in the first month, both from "obviously safe" prompt edits. The harness pays for itself every time it turns red.

Lessons Learned

1. Optimize for adoption first, coverage second.
2. Grade with a rubric, not vibes.
3. Fail the build — a warning nobody reads is not a test.