How to Evaluate Coding Agents: Benchmarks, Trajectories, and Where Scores Lie

Coding agent evaluation is where most teams get fooled. A vendor posts a shiny SWE-bench number, an engineering lead treats it as a purchase decision, and three weeks later the agent is quietly opening broken pull requests against a real codebase. The leaderboard was true and useless at the same time. If you are deciding which coding agent to trust — and how to prove it works on your code — this guide moves you from single-number thinking to a durable evaluation method that survives contact with production.

The short version: aggregate benchmarks tell you whether an agent can plausibly write code, not whether it will help your team. To answer that, you have to look at how the agent got to its answer, not just whether the final answer passed a test.

Why do coding benchmarks mislead?

Benchmarks are compressed. They take thousands of hours of messy, context-heavy engineering work and squeeze it into a pass/fail rate. That compression is exactly what makes them portable — and exactly what makes them lie by omission.

OpenAI's own write-up on the problem, Separating signal from noise in coding evaluations, makes the core point plainly: a lot of what moves a coding score is noise, not capability. Run-to-run variance, prompt sensitivity, and the harness around the model can swing results enough that two "improvements" are statistically indistinguishable. If you cannot separate the signal from that noise, a higher number is not evidence of a better agent.

Three failure modes explain most of the gap between a leaderboard and reality:

Contamination. If a benchmark's tasks or their solutions leaked into training data, the score measures recall, not problem-solving. A model can look brilliant on a public benchmark and mediocre on the private code it has never seen.
Harness bleed. The scaffolding around the model — retries, tool access, context assembly, how a task is even presented — can dominate the result. You may be benchmarking the harness, not the model. That is not a footnote; a recent arXiv paper, The Harness Effect, argues orchestration design is significant enough to set the token economics of enterprise agentic AI, meaning the same underlying model can be cheap or ruinously expensive depending on how it is wired up.
Metric myopia. pass@1 rewards getting the final diff right. It says nothing about how many wrong turns, wasted tokens, or unsafe edits happened along the way — the things that actually cost your team time and money.

The refresh cadence of benchmarks themselves is a tell. When Google revamped its Android AI development benchmark — adding new agents and finding Gemini still lagging — it was a reminder that benchmarks are moving targets maintained by interested parties. A score is a snapshot of one harness, on one task set, at one moment.

What does a trajectory-level evaluation actually inspect?

If the final diff is the least informative artifact, the trajectory — the full sequence of steps the agent took to produce it — is the most informative. Trajectory evaluation asks: how did the agent reason, which tools did it call, where did it backtrack, and did it do anything dangerous on the way to a passing test?

This is the shift described in the arXiv work AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation, which frames evaluation around reviewing real trajectories rather than only scoring end states. The practical inspection checklist that follows from a trajectory-first mindset:

Path efficiency. Did the agent reach the fix in a few deliberate steps, or did it flail — editing, reverting, re-editing the same file? Flailing that happens to land on a passing test is not a win you can rely on.
Tool discipline. Which tools did it invoke, and did it read before it wrote? An agent that greps and reads context before editing behaves very differently from one that pattern-matches blindly.
Blast radius. Did it touch only what the task required, or did it rewrite unrelated files to make a test go green? Collateral edits are how agents pass benchmarks and fail code review.
Failure honesty. When it could not solve the task, did it stop and say so, or did it fabricate a plausible-looking diff? Graceful failure is a feature.

None of these show up in a pass rate. All of them show up in a trajectory. This is also why coding agent observability — capturing and being able to replay what the agent did — is becoming part of the eval stack, not just the ops stack.

How do you build an eval harness that mirrors production?

The most trustworthy benchmark is the one you build from your own repository. A practical, source-informed harness looks like this:

Curate tasks from your real backlog. Pull recently closed issues and PRs from your own codebase and turn them into eval tasks. Because they never appeared in any public dataset, contamination is off the table by construction.
Fix the harness before you compare models. Given the harness effect, hold orchestration, tool access, and context assembly constant across every agent you test. Otherwise you are comparing scaffolds, not agents.
Score more than pass/fail. Record task completion and the trajectory metrics above: steps taken, tokens spent, files touched, and whether the change would survive human review. This is where you catch the agent that passes the test but doubles your token bill — the economic dimension the Harness Effect paper puts front and center.
Run it repeatedly and report variance. One run is an anecdote. Following OpenAI's signal-vs-noise guidance, run each task multiple times and report the spread. If two agents' ranges overlap, you do not yet have a winner.
Review trajectories, not just totals. Sample the actual paths — especially the failures. This is the AgentLens lesson: production-assessed trajectory review surfaces problems that aggregate scores hide.

FAQ

How do I evaluate a coding agent for my own codebase?

Build a small, private benchmark from your closed issues and merged PRs, hold the harness constant across candidates, and score both outcome (did it work) and trajectory (how it worked, at what token cost). Run each task several times and compare the ranges, not single scores. This beats any public leaderboard because it is contamination-proof and reflects your real code.

Are SWE-bench and other public coding benchmarks trustworthy?

They are useful as a coarse floor — a model that fails them broadly is unlikely to help you — but untrustworthy as a purchase decision. Public scores are vulnerable to contamination, dominated by harness design, and reported without variance. Treat them as a screening signal, then validate on private tasks.

What is trajectory evaluation and why does it matter?

Trajectory evaluation reviews the full sequence of steps an agent took — its reasoning, tool calls, backtracks, and edits — rather than only the final diff. It matters because two agents can produce the same passing result while one worked cleanly and the other flailed, over-edited, or got lucky. The trajectory is where reliability and cost actually live.

Why does token cost belong in a coding-agent evaluation?

Because orchestration design can make the same model cheap or expensive, an agent that reaches the right answer inefficiently can cost far more per resolved task. Measuring tokens-per-completed-task alongside accuracy keeps you from adopting an agent that is technically capable but economically painful.

Takeaways for Clawvard readers

Distrust single numbers. A benchmark score is a snapshot of one harness on one task set — a screening signal, never a verdict.
Fix the harness, then compare. Orchestration can dominate results and set your token bill; hold it constant before ranking agents.
Evaluate trajectories, not just outcomes. How the agent got there predicts reliability better than whether the test went green.
Build your own private benchmark. Tasks from your real backlog are contamination-proof and the only evaluation that measures the agent on the code you actually ship.

Want to go deeper on the infrastructure side of running agents well? Read our companion piece, Remote MCP and Managed Agents, Explained, and see how Clawvard helps teams evaluate and orchestrate coding agents without getting fooled by a leaderboard.