Deterministic checks beat model-judged evals in CI.

An agent can complete its task and still fail you. While building agent-reliability-harness, I kept running into the same category of bug: the trace looks successful, the user got an answer, and something in the middle was still wrong. A tool was called that the workflow never approved. A required argument was missing and the tool silently coped. A response that needed evidence shipped without a single citation.

The popular fix is LLM-as-judge: hand the trace to a second model and ask whether the agent behaved. I think that is the wrong default for CI, and I want to argue for something more boring.

The problem with a judge in the merge path

A model judge in CI has three costs that compound. It is flaky: the same trace can pass on Monday and fail on Tuesday, which trains a team to re-run the pipeline instead of reading the failure. It is expensive: every push burns tokens on questions that mostly have mechanical answers. And it is unexplainable in the way that matters for debugging: “the judge scored this 6/10” does not tell you which step to fix.

None of that means judges are useless. It means the merge path is the wrong place for an opinion.

What I gate on instead

The harness validates agent execution traces against a versioned policy file. Four families of checks, all deterministic:

• Tool-call contracts. Each allowed tool declares required and optional arguments with types. Unapproved tools, missing arguments, wrong types, and undeclared arguments all fail loudly.
• Budgets. Maximum total latency, per-step latency, and total cost. An agent that answers correctly but blows the cost envelope is still a failing agent.
• Safety patterns. Case-insensitive regexes for content that should never appear in a tool argument, response, or tool output: prompt-injection phrases, password-like strings.
• Grounding coverage. Responses marked as requiring evidence are measured for citation coverage against a policy threshold.

The output is a weighted 0–100 score per trace, and the failures are named:

[PASS] lead-qualification-0001  score=100.0/100
[FAIL] renewal-workflow-0007    score=80.0/100
[FAIL] support-escalation-0042  score=70.0/100

SUMMARY: 1/3 traces passed, average score 83.3/100

When support-escalation-0042 fails, the report says why: an unapproved tool, missing citations, and an injection phrase in a tool output. That is a diff-sized fix, not an investigation.

Why determinism is the feature

Same trace, same policy, same score, forever. That single property buys everything else: failures are reproducible, so they are debuggable. Checks are free, so they run on every push. Policies live in JSON, so tightening a budget or approving a tool is a code-reviewed diff with an author and a reason. The policy file becomes the honest, versioned record of what the team considers acceptable agent behavior.

I also kept the runtime standard-library only. No framework dependency means adopting it in CI is one install and one command, and there is no second agent stack to maintain inside your test infrastructure.

Where judges still belong

Offline, on samples, for the qualities that genuinely resist rules: tone, helpfulness, whether an answer actually addressed the question. Run a judge nightly over a sampled slice and review the trend line. Just do not let an opinion block a merge.

Honest limits

Regex safety checks are tripwires, not guarantees; they catch known-bad patterns and nothing else. Schema checks catch drift, not intent. And a 100/100 trace can still be a bad user experience. Deterministic gates do not replace evaluation; they are the floor that makes everything above them cheaper to build.

The code, sample policies, and failing traces are in the repository. If you gate agents differently in CI, I would genuinely like to hear how: hello@felmon.tech.