Agent evaluation and reliability

A model that is right 95% of the time feels excellent. An agent that needs twenty such steps in a row succeeds 36% of the time — and that single piece of arithmetic explains most of the gap between agent demos and agent products. This closing lesson of the track is about living with that exponent: measuring agents at the level they actually fail (trajectories, not steps), engineering against compounding, and the containment discipline that lets you ship a system that acts — the final form of every safety idea this platform has taught, from its own honeypot to the IAM lesson’s worst-hour question.

The compounding law

If each step of a task succeeds independently with probability $p$, a $k$-step task succeeds with $p^k$ — and the exponent is merciless:

The independence assumption hides the second lesson: real agent errors correlate (a misread requirement poisons every subsequent step), so $p^k$ is often optimistic — but correlated failure also means one good verification step can catch whole families of error at once. That asymmetry is why checks are the highest-leverage steps in any loop.

Evaluating the trajectory, not the step

The LLM track’s evaluation lesson graded outputs; agents need grading of behavior. The working toolkit:

• Outcome evaluation: did the task end in the goal state? Wherever the environment can verify (tests pass, order actually refunded, file actually parses), outcome checks are gold — cheap, objective, and the reason coding agents advanced fastest: their world grades itself. The design corollary runs backward: make your agent’s tasks verifiable and you have manufactured your own training signal.
• Trajectory evaluation: two agents that both succeed are not equal if one took 6 steps and one took 60 — cost, latency and side effects are metrics; an LLM judge reviewing the transcript (“was every tool call justified? did it recover from the error or luck past it?”) is the judge-noise lesson’s machinery pointed at process, with all the same calibration caveats and pair-count arithmetic.
• Capability vs reliability, reported separately: pass@1 vs pass@8 (can it ever, vs does it usually) measure different products. A 60% pass@1 / 95% pass@8 agent is shippable with retries; the same headline number without the split is marketing.
• Benchmarks with eyes open: SWE-bench-style suites made agent progress legible, and inherit every contamination and Goodhart caveat the evaluation lessons taught — plus a new one: agents can pass by exploiting harness quirks, so transcript audits of passing runs are part of honest benchmarking, not paranoia.
• Staged rollout is evaluation: offline suite → shadow mode (agent acts, effects discarded, decisions compared) → human-approval mode → autonomy with sampling audits. The promotion ladder from the CI/CD lesson, applied to trust.

Containment: engineering for the failures you didn’t foresee

Reliability work assumes known failure modes; containment bounds the unknown ones. The stack, every layer already taught somewhere in this curriculum, now composed:

1. Least-privilege tools — the agent’s tool set is its blast radius (IAM lesson); read-only by default, mutating actions behind draft-and-approve.
2. Budgets — turns, tokens, dollars, wall-clock; the runaway scenario’s defense, and the difference between “agent failed” and “agent failed expensively at 3 a.m.”.
3. Irreversibility gates — actions sorted by undo-ability; the irreversible ones (send, delete, pay) require confirmation structurally, not by prompt-side pleading, because prompts are advice and gates are architecture.
4. Audit trails — every tool call logged with arguments and results (CloudTrail for cognition); the transcript is both your debugging artifact and your evaluation corpus.
5. Monitoring for behavioral drift — tool-call distributions, loop lengths, refusal rates over time; agents change when models, tools or the world change, and the platform’s own honeypot exists precisely because other people’s agents are already out there acting.

Close the track’s arc: pathfinding taught search with guarantees; RL taught learning without them; reasoning loops taught inference-time self-improvement; agents put all of it behind an API key and a tool set. The mathematics got less certain at every step while the consequences got more real — which is why this final lesson is mostly engineering discipline, and why the discipline is not optional equipment. Autonomy is earned in production the same way trust is earned anywhere: bounded first, audited always, expanded on evidence.

import numpy as np

# the compounding law and its two fighters, in ten lines
p, k = 0.95, 20
print(f"bare:        {p**k:.1%}")                       # 35.8%

p_retry = 1 - (1 - p) ** 2                              # one retry per step
print(f"with retry:  {p_retry**k:.1%}")                 # 95.1%

# checkpoint blocks of 5, block retried up to 3x
block = p_retry ** 5
block_ok = 1 - (1 - block) ** 3
print(f"+ checkpoint blocks: {block_ok**(k//5):.3%}")   # 99.999%

# and the inverse question every roadmap needs:
for target, steps in [(0.9, 20), (0.9, 50)]:
    print(f"want {target:.0%} at k={steps}: need p = {target**(1/steps):.3%}/step")

Exercises