Applied AI reliability
Agent Reliability Harness
- Problem
- An agent can return a perfect-looking final answer while double-issuing refunds, skipping checks, or leaking secrets along the way.
- What I built
- A trajectory-regression harness on PyPI: policy-as-code trajectory rules, baseline-vs-candidate CI gates, OpenAI/Anthropic transcript adapters, JUnit/SARIF reports.
- Technical decision
- Zero runtime dependencies and byte-deterministic reports, so the evaluator itself can never flake a CI pipeline.
- Verified result
- v0.2.1 on PyPI; 194 tests across 3 OSes and Python 3.11-3.13; benchmark precision/recall 1.0 on 34 seeded failures; a live demo PR blocked by the gate.
- What I learned
- Deterministic policy checks beat model-judged evals in CI: cheaper, explainable, and reproducible.
- Python
- Agent evaluation
- GitHub Actions
- PyPI