I Created a Coding Harness Without Realizing It

I didn't set out to build a coding harness. I set out to learn Mastra.

If you haven't run into it yet, Mastra is an open-source TypeScript framework for building AI applications — agents, workflows, tools, memory, evals, all the pieces. I'd been living inside Claude Code for months, and I'd already done one round of encoding my judgment into that world: a repo full of plugins, hooks, and commands that captured how I think software should get planned, reviewed, built, and shipped. It worked. But it was judgment expressed in one vendor's configuration format.

So the Mastra project started as a learning exercise. Build a few agents. Wire up a workflow. See what the framework felt like.

I also brought my Cloudflare opinions with me, because I wasn't interested in building a generic app builder. I build standalone Cloudflare Workers. Vanilla HTML, CSS, and JavaScript. Wrangler, D1, KV, R2. No React, no build steps, no framework du jour. If I was going to have agents write code, they were going to write my kind of code, on my stack.

Then something happened that I didn't fully see until I stepped back.

Every time an agent did something I didn't like, I didn't write a nicer prompt. I wrote a rule. When the rule kept mattering, I wrote code that enforced it. When an agent claimed it finished something, I stopped reading the claim and started checking the file system. When a loop ran too long, I gave it a budget. When quality was fuzzy, I wrote a rubric. When the rubric needed to stay honest, I gave it test cases.

One day I looked up and realized what I'd actually built. Not "some agents and a workflow." A harness — the whole apparatus that surrounds AI models and keeps them safe, honest, and productive. Planning, architecture review, a build loop, a release gate, deployment approval. Deterministic checks, judges, scorers, eval suites running in CI. An event log recording every action every agent ever took.

Let me be clear about what this isn't: I'm not competing with Codex or Claude Code. Those are general-purpose tools built by teams of brilliant people. Mine is a personal factory, narrow on purpose. It builds one kind of thing — small Cloudflare Worker apps from a plain-English vision doc — the way I would build them.

But because it's narrow and personal, it's an unusually clean look at what a harness actually is. The whole thing is open source: github.com/chrislema/mastra-builder.

Along the way, I stuck to my guns about what matters. Here are the seven things I'd tell anyone building with AI agents — with just enough detail from the project to make each one concrete.

1. Sort every rule into its home

This is the organizing principle underneath everything else. Every rule in the system lives in exactly one of three homes.

Deterministic and blockable? It goes to code. Is the task dependency graph free of cycles? A graph algorithm answers that in a millisecond, for free, the same way every time. No model gets a vote.

Judgment, but gradeable? It goes to a rubric. Is this plan concrete or just themes? Are these acceptance criteria actually checkable? A judge model scores it against a written standard — and plain TypeScript does the aggregation math, because a model that computes its own final grade can talk itself into a pass.

Judgment and generative? It goes to an agent. Writing new code, decomposing a vague goal, reviewing architecture. The genuinely creative work is the only work that should cost you model tokens.

And here's the part people miss: rules migrate down. My planner kept escalating preferences as "blocking questions." At first, that was prompt guidance ("ask only blocking questions"). When the mistake kept repeating, it became a coded gate that rejects any open decision that isn't decision-shaped and doesn't name what it blocks. Judgment that repeats itself is judgment you should stop paying for.

2. Claims are not evidence

An LLM is a confident narrator, not a reliable one. It will tell you the tests passed. It believes it. Sometimes it's wrong.

So my harness never trusts self-reporting. Every tool call, file write, command, and deployment lands in an append-only event log. When an agent finishes a task, the "implementation note" isn't written by the agent — it's synthesized from the log. Files touched come from recorded write events, not from what the agent says it touched. Verification results come from commands the workflow itself ran.

There's even a guard on the way out: if an agent's response claims completion ("done," "implemented successfully," "deployment complete") without citing any actual evidence — artifact paths, event references, test output — the response is rejected and retried with instructions to cite its receipts. The harness will not accept "trust me" as an answer, structurally.

3. Constrain, don't implore

Early on, everyone writes prompts like "please don't modify files outside your task." Here's the thing: a paragraph asking nicely is a suggestion. A hook that blocks the write is a law.

In my system, every workspace action passes through a hook before it executes. The designer agent physically cannot edit a database migration — the write is refused mid-keystroke, with a reason, and the refusal is logged. Dangerous shell commands are blocked by a list, not discouraged by prose. If any agent tries to write code using bcrypt — banned by my security policy — the write never reaches the disk. There's even a read budget: six investigative reads before writing anything, and further reads are refused, because AI coders love to wander through a codebase while the token bill climbs.

Spend your prompts on judgment. Spend code on rules. Every rule you move from prompt to hook is obedience you no longer have to hope for.

4. Bound every loop; make "stuck" a first-class state

Loops are where agent systems get good — try, check, retry — and also where they burn money thrashing. Two disciplines fixed this for me.

First: every loop has a budget. Plan repair, architecture review bounces, build attempts, release-gate retries — all capped. When the budget runs out, the task doesn't limp forward and doesn't spin. It parks as stuck, downstream work gets marked blocked, and the run stops with a clear record of what failed and what fix was suggested. A line from my project's constitution: a bounded loop that parks as stuck beats an unbounded loop that thrashes. Stuck isn't the system failing. Stuck is the system working.

Second: retries are diagnosed, not repeated. When an attempt fails, code classifies the failure — never created the files? left placeholder stubs? specific compiler errors? — and reshapes the next attempt accordingly. A "never wrote anything" failure gets retried with only write tools and an order to create the missing files immediately. A verification failure gets retried with the exact compiler diagnostics and current file contents in hand. Each retry is narrower and cheaper than the attempt before it.

5. Fail closed, everywhere

When something is ambiguous, the safe answer wins. That has to be the default at every layer, because doubt shows up everywhere.

Tester returns release-gate output that won't parse? The code synthesizes a failing gate — every critical area marked missing, deployment stopped. A rubric gate that nobody evaluated? Failed, with the evidence string "gate was not evaluated — failing closed." The judge model's provider has an outage mid-run? The judgment fails closed with a note that clearly says the judge was unavailable — retry the run, nothing is implied about your code — because infrastructure failure should never be mistaken for quality failure in either direction.

Garbled is not "probably fine." Unproven is not "passing." The one posture that makes an autonomous system trustworthy is that it treats silence and confusion as red lights, not green ones.

6. Measure the measurers

Here's the question that separates a demo from a system: you tweaked a prompt, or swapped in a cheaper model. How do you know you didn't just make everything dumber?

Traditional code has tests. AI judgment needs the equivalent, at every level. In my project: every rubric embeds a known-good and known-bad exemplar, and a rubric that can't tell its own exemplars apart isn't allowed to be used. Every scorer is covered by a dataset with ground truth — including negative cases, because a suite of happy paths can't catch a scorer that quietly started saying yes to everything. And the whole thing runs as a gate in CI: a change that degrades the system's judgment fails the build, with a machine-written explanation of exactly which fixture disagreed.

This is what lets you experiment cheaply. I have the execution agents set up to swap to a cheaper model with one environment variable — and when I do it, the evals tell me what happened. Numbers, not vibes.

7. Interrupt humans only for genuine blockers and irreversible actions

An autonomous system that constantly asks permission isn't autonomous. And a human who gets interrupted for preferences stops reading the interruptions.

My workflow can pause and wait for a human at exactly two moments. One: the planner hits a true blocker in the source documents — and it works hard not to, filtering out settled-policy questions and anything resolvable by a safe default, which gets recorded as a visible, checkable assumption instead. Two: production deployment. The default run mode never touches production at all; the real deploy requires an explicit, logged human approval before the command runs.

Everything else either proceeds on a recorded assumption or parks as stuck for later review. Escalation is a budget. Spend it on the two things that deserve it: questions only a human can answer, and actions no one can undo.

The kicker

Here's the part I didn't plan.

After all of this was built, I picked up the two books the Mastra team wrote — Principles of Building AI Agents and Patterns for Building AI Agents. I hadn't read them during the build. Within an hour of getting them, I distilled them into a 100-point evaluation rubric for AI agent projects, and then scored my own repo against it.

It scored 90 — and the ten missing points landed precisely on gaps my own docs had already confessed to.

Two independent derivations — one from a practitioner grinding on a real project, one from books I'd never opened — converging on the same principles, the same weights (evaluation is the heaviest category in both), and the same unfinished work. That's usually the sign that both are describing something true about the territory rather than about each other.

I didn't set out to build a harness. But it turns out that if you refuse to trust claims, refuse to beg when you can block, refuse to let loops run wild, and insist on measuring everything that judges — a harness is just what that refusal looks like, written down in code.

The whole thing is here: github.com/chrislema/mastra-builder. Steal what's useful.