CodyCody
← All posts

Agent reliability

Claude Code's Generated Workflows Are Code Nobody Tests

A workflow is a program, and a program nobody has run more than once is untested by definition — true whether a person wrote it or a model did. Claude Code's dynamic workflows generate exactly that kind of program: trigger one, and Claude writes a fresh JavaScript script for the task you just described, then a runtime executes it immediately. There's no gap between the script being written and the script being trusted with your task — the first run of the workflow is the first run of the code.

That's fine until the input changes. Trigger a dynamic workflow to lint every changed file and open a PR only if it's clean, and the fan-out loop that comes back assumes at least one file. The first run, there are three, and it works. The fourth run, there are zero, and it either throws or — worse — reports the stage as clean without ever running it. Nobody wrote a test for the empty case, because the script never existed in a state where writing one was possible: generated and executed in the same breath.

Only tested code earns trust — and this skips the test by construction

Ordinary code earns trust by being run against more than one input, reviewed by someone other than its author, and fixed in place when a bug turns up, with the fix staying fixed next time. A dynamic workflow's script skips all three by design, not neglect: generated for one task, run once, judged done the moment it produces a plausible result for that one input. Anthropic's own framing says as much — a script "Claude writes for the task you describe." Saving it as a reusable /command freezes whatever it already does, bugs included.

It's also why the feature's real safeguard — rerunning a stage that fails outright — doesn't close the gap. A rerun only fires on a thrown error. It does nothing for the more common bug: a stage that completes and returns the wrong value for an input its author didn't picture, no exception raised. And because each session gets its own generated script, a fix made in one run doesn't carry forward — the next similar task is a new script.

What actually closes the gap

Testing and optimizing both need a target that stays still long enough to run more than once, or to be refined against what the last run got wrong — a procedure that's the same object every time, authored once and executed unchanged until someone deliberately edits it. CodyCody's engine works this way: a workflow is designed once and stored on the engine rather than regenerated per session, its steps' output captured and hashed so a later step can check it against reality. A fix stays verifiable — rerun the exact procedure that broke and confirm it holds, on run four and every run after.

This isn't only a problem for complicated workflows — a two-step script has an empty-input case as easily as a twenty-step one. It's just less likely anyone notices.

Related: why coding agents skip steps and how to actually verify what an agent did cover the agent's own account of a run; this one is about the procedure it's running being untested in the first place. See an authored-once procedure in practice in a verified code-review workflow. CodyCody has a free Starter tier — see pricing.

Disagree, or hit this in practice? Discuss it on the forum.