CodyCody
← All posts

Agent reliability

How to Actually Verify What Your AI Coding Agent Did

"Trust but verify" isn't a policy until you've picked a mechanism. Below are the five that actually exist for a Claude Code or Codex session, ranked by what each one actually checks and what it costs you to keep running. None of them are wrong to use — they compose — but they verify different things, and knowing which is which matters more than picking one.

1. Read the diff yourself

The baseline, and still necessary no matter what else you add. It catches the end state: does the code look right. It catches nothing about execution — a plausible-looking change and a change that was actually tested against a passing suite produce the same diff. This is where fabricated test results live undetected: the diff has no opinion on whether the tests behind it were real.

Who maintains it: you, every time, by hand. Doesn't scale past a handful of changes a day.

2. Ask the agent to paste real terminal output inline

Better than a bare claim of success, and free. Still self-reported: nothing stops the pasted block from being summarized, cleaned up, or — per the mechanism in the article above — generated as a plausible block rather than copied from a real run. It raises the cost of fabricating a result slightly. It doesn't remove the option.

Who maintains it: you, by remembering to ask, every time.

3. A PreToolUse or PostToolUse hook

The one native Claude Code mechanism with real teeth: a PreToolUse hook that exits with code 2 blocks a matching tool call "regardless of what Claude decides" — enforcement in the harness, not a request to the model. A PostToolUse hook can log a command's actual output to a file you control, independent of anything the agent later says about it.

What it doesn't do on its own: compare the two. A log is a fact sitting next to a claim; nothing forces them to be checked against each other unless you write that check yourself, one hook script at a time, and keep it current as your procedure grows.

The gap a hook closes isn't hypothetical, and it isn't only about trusting the agent's own reports. The August 2025 "s1ngularity" supply-chain attack against the Nx build system — malicious npm packages live for roughly five hours, hitting a project with 4.6 million weekly downloads — worked in part by weaponizing exactly the permission flags a hook is built to gate: the malware "weaponized installed AI CLI tools by prompting them with dangerous flags (--dangerously-skip-permissions, --yolo, --trust-all-tools) to steal file system contents," turning Claude Code, Gemini CLI and Amazon Q already installed on a victim's machine into reconnaissance tools against their own credentials. Over 2,349 secrets were stolen. A PreToolUse hook that refuses to let those flags reach a tool call is a real, available defense — the attack succeeded specifically where nothing was gating them.

Who maintains it: you, in shell scripts, indefinitely.

4. A dynamic workflow's rerun-on-failure

Claude Code's newer dynamic workflows hand step order to a script rather than to the model turn by turn, and a failed stage gets rerun rather than silently accepted. That's real, script-level determinism for the run it covers.

The gap: the script itself has no shell or filesystem access — the subagents it spawns do that work, under ordinary tool permissions, not a captured-and-rehashed contract. And Anthropic's own docs are direct that there's no built-in mid-run human gate: "for sign-off between stages, run each stage as its own workflow." A rerun catches a stage that errored outright. It doesn't catch a stage that reported success without actually doing the work.

Who maintains it: Claude writes the script per task; you maintain it if you save one for reuse.

5. An exit-code-and-hash contract, checked by a later step

This is where CodyCody's shell steps sit, and it's the only one of the five where the check doesn't depend on remembering to add it. The engine fixes the command; the agent runs it under a structured contract — exit code, output, and a sha256 of that output captured to a file; a later step in the same workflow reopens the file and recomputes the hash itself, rather than trusting the number the agent reported.

Said plainly, once more: this doesn't make a fabricated report impossible at the instant it's made. It makes the report re-checkable by something that wasn't told what answer to expect, at a point the agent doesn't control. A fabricated hash doesn't match a file that was never actually written to, and the mismatch is what fails — not a claim measured against another claim.

Who maintains it: the engine. You author the procedure once; every run since gets the same check.

Which one to reach for

Layer 1 is not optional — always read the diff. For most day-to-day work, layer 3 (a couple of well-placed hooks) is a real, free upgrade with no infrastructure to run. The case for layer 5 grows with the size of the procedure: the more steps an agent runs unattended, and the higher the cost of one silent skip, the more that "maintained by you, one script at a time" in layers 3 and 4 turns into a workflow engine you've built by hand, badly, over several months.

Worth asking before you pick one

Do I need a workflow engine for a small project? No. A handful of tasks a week is well served by diff review plus a hook or two. The math changes once the agent is running a multi-step procedure unattended — a release, a migration, a review — often enough that maintaining the checks yourself becomes its own job.

What if I just ask the agent to double-check its own report? That's still the model grading its own work — the same self-report the original claim came from, checked by the same process that produced it. It doesn't add an independent check; it adds another sentence of the same kind.

Does more verification slow the agent down? A little, the way a test suite slows down a commit. The trade is the same one: a few extra seconds per step against a failure that would otherwise reach production looking like a success.

Related: why coding agents skip steps and why they get worse on long tasks — both are versions of the same gap this post is about closing. See the contract in practice in a verified code-review workflow. CodyCody has a free Starter tier — see pricing.

Disagree, or hit this in practice? Discuss it on the forum.