AI Agent Evaluation: How to Test and Score Agents in 2026

How to test an agent instead of just watching it fail in production

Most teams ship their first agent on manual testing and gut feeling. That works until the agent is live and someone asks why it got worse after last week's prompt change, and there is no record to check against. AI agent evaluation is the discipline that replaces the guess with a repeatable test, and it looks nothing like grading a single chatbot reply.

AI agent evaluation is the practice of giving an agent a defined task, letting it run its full tool-calling loop, and then scoring either its final output, its complete trajectory of reasoning and tool calls, or one decision inside that trajectory, using code-based, model-based or human graders. It exists because agents fail in more places than a plain LLM call does.

How it works

Anthropic's engineering team, which builds evals for Claude Code and for customers building agents at scale, breaks the vocabulary down into specific pieces. A task is a single test with defined inputs and success criteria. Each attempt at a task is a trial, run more than once because model outputs vary. A grader is the logic that scores some part of the agent's performance, and a task can carry several graders each with multiple assertions. A transcript (also called a trace or trajectory) is the complete record of a trial: every tool call, every intermediate result, every piece of reasoning. The outcome is the actual final state in the environment, not what the agent claims happened, so a flight-booking agent that says "your flight is booked" only passes if a reservation actually exists in the environment's database. The evaluation harness is the infrastructure that runs all of this end to end, distinct from the agent harness (or scaffold) that lets the model act as an agent in the first place.

Agent evaluations combine three kinds of graders, and picking the wrong one for the job is where most eval suites go wrong, according to Anthropic's breakdown:

• Code-based graders: exact or fuzzy string matches, fail-to-pass and pass-to-pass binary tests, static analysis (lint, type, security), tool-call verification, and transcript analysis of turns taken or tokens used. Fast, cheap, objective and reproducible, but brittle against valid answers that do not match the expected pattern exactly.
• Model-based graders: rubric-based scoring, natural-language assertions, pairwise comparison, reference-based evaluation and multi-judge consensus. Flexible and able to handle open-ended output, but non-deterministic and more expensive to run at scale.
• Human graders: still used for calibration and for the subjective judgment calls a rubric cannot fully encode.

DeepEval, the open-source LLM evaluation framework, splits agent evaluation into three scopes that determine which metric applies. End-to-end (black-box output) evaluation looks only at the agent's final output. Trajectory-based evaluation scores the complete ordered execution trace, including reasoning, tool calls and intermediate steps. Component-level (step-wise) evaluation scores a single span in isolation, such as the specific LLM call that picked a tool and generated its arguments. DeepEval maps these scopes to a three-layer model of an agent: the reasoning layer (plans and strategy, scored with its PlanQualityMetric and PlanAdherenceMetric), the action layer (tool selection and argument generation, scored with ToolCorrectnessMetric and ArgumentCorrectnessMetric), and the execution layer (the full loop, scored with TaskCompletionMetric and StepEfficiencyMetric). PlanQualityMetric uses an LLM judge to score how well an extracted plan addresses the task; if the trace shows no explicit plan, it passes by default with a score of 1. PlanAdherenceMetric then checks whether the agent actually followed the plan it made, since a good plan the agent ignores mid-execution is its own kind of failure.

Examples

Descript, whose agent helps users edit video, built its evals around three dimensions: don't break things, do what I asked, and do it well. The team evolved from manual grading to LLM graders with criteria written by the product team and checked periodically against human calibration, and now runs two separate suites, one for quality benchmarking and one for regression testing, according to Anthropic's write-up. Bolt's AI team took the opposite path: they built their agent first and added evaluation later, once it was already widely used. In three months they built a system that runs the agent, grades output with static analysis, uses browser agents to test the resulting apps, and applies LLM judges for behaviors like instruction following.

Anthropic's own Claude Code followed the same later-stage pattern: it started with fast iteration on feedback from employees and external users, then added evals first for narrow, easy-to-check areas like concision and file edits, and only later for harder behaviors like over-engineering. Those evals now run alongside production monitoring and A/B tests rather than replacing them.

The benchmark side of agent evaluation has its own case study in τ2-bench, built by Sierra to test tool-agent-user interaction in retail, airline and telecom domains, with dual control so the simulated user can also act on the shared environment. Anthropic reported that Claude Opus 4.5 was scored against a τ2-bench flight-booking task, found a loophole in the stated policy, and used it to book a better outcome for the user than the intended solution allowed. The eval marked the run a failure because the transcript did not match the expected policy path, even though the agent had, in a real sense, done its job better than the test anticipated. That is the general risk with rigid code-based graders on capable models: they can penalize a valid, creative solution because it does not match a fixed expected answer.

When to use it (and when not)

Manual testing, dogfooding and intuition get a team surprisingly far in early prototyping. Building a full eval suite before an agent has real users can be overhead that slows shipping down for no reason. The pattern shows up reliably once an agent moves past prototyping and starts scaling: users report the agent "feels worse" after a change, and there is no way to check except to reproduce the complaint by hand and hope nothing else broke in the process.

That is the point to invest in evals: before a change ships to production, to catch regressions before users do; when adopting a new model, since teams with an eval suite can benchmark a new model's strengths and update prompts in days while teams without one spend weeks re-testing by hand; and to get cost, latency and token-usage baselines for free on a fixed bank of tasks. Evals are also useful earlier than that, at the start of development, because writing them forces the team to specify what success actually means, resolving the ambiguity that two engineers reading the same spec would otherwise interpret differently.

What's new (as of 2026-09-13)

DeepEval's 2026 changelog logged a breaking change: every metric score in the framework is now higher-is-better, reversing the direction some scores previously used. The same changelog lists richer tracing fields and OpenTelemetry export support, expanded frontier and provider model entries with more accurate pricing, cleaner component-level evaluation with active-trace assertions and structured result exports, and new controller APIs and custom templates for multi-turn conversation simulation.

On the benchmark side, Sierra's τ-bench family kept adding domains through 2026: τ²-bench (June 2025) introduced dual control, letting the simulated user act on the environment directly instead of only the agent, and added a telecom domain. τ³-bench (March 2026) added banking, including τ-knowledge tasks where agents retrieve and reason over a roughly 700-document knowledge base, and τ-voice, a real-time voice version testing full-duplex conversations with interruptions, accents and background noise. Sierra also audited and fixed more than 50 existing airline and retail tasks in February 2026, correcting ambiguous instructions and impossible constraints in the original benchmark.

As of 13 September 2026, the τ2-bench leaderboard tracked by Price Per Token, sourced from Artificial Analysis, put Z.AI's GLM-5.2 in first place at a 99.1% score, ahead of GLM-4.7-Flash Thinking at 98.8% and GLM-5 Turbo at 98.5%. Anthropic's Claude Fable 5 also scored 98.5% on the same leaderboard, which had evaluated 233 models with an average score of 52.0.

Key takeaways

• AI agent evaluation scores a task, a trial, and a grader against a transcript and an outcome, not just a single input-output pair like a conventional LLM eval.
• Code-based graders are fast and objective but brittle to valid answers that do not match an expected pattern; model-based graders handle open-ended output but are non-deterministic and costlier to run.
• DeepEval's three evaluation scopes, end-to-end, trajectory-based and component-level, map to an agent's reasoning, action and execution layers, and each needs a different metric.
• Descript's agent evals cover three dimensions (don't break things, do what I asked, do it well); Bolt built its eval system in three months using static analysis, browser agents and LLM judges.
• On the τ2-bench leaderboard as of 13 September 2026, GLM-5.2 topped the field at 99.1% out of 233 evaluated models.

FAQ

What is AI agent evaluation?

AI agent evaluation is the process of giving an agent a defined task with success criteria, running its full tool-calling loop, and scoring the result with code-based, model-based or human graders, applied to either the final output, the complete trajectory, or a single decision inside it.

What are the three types of graders used to evaluate AI agents?

Code-based graders (string matching, static analysis, tool-call verification), model-based graders (rubric scoring, LLM-as-judge, pairwise comparison) and human graders. Anthropic's guidance is to combine them rather than rely on one, since each covers weaknesses the others have.

What is a trajectory in agent evaluation?

A trajectory, also called a transcript or trace, is the complete record of one attempt at a task: every tool call, every intermediate result and all of the agent's reasoning, not just its final answer. DeepEval's trajectory-based metrics, like PlanQualityMetric and TaskCompletionMetric, score this full record rather than the output alone.

Why did a correct agent action still fail an evaluation?

It happens when a code-based grader checks for one expected path and the agent finds a different, valid one. Anthropic documented Claude Opus 4.5 finding a loophole in a τ2-bench flight-booking policy that produced a better outcome for the user but did not match the test's expected trace, so the eval marked it a failure despite the better real-world result.

When should a team start building agent evals instead of testing by hand?

Manual testing and dogfooding work during early prototyping. The signal to build a real eval suite is when an agent is in production and a team cannot tell a real regression from noise after a change, or when adopting a new model requires re-testing everything by hand instead of running it against an existing benchmark.

Teams comparing tools that trace and evaluate their own agents can browse the current options on the AI Agents Listing monitoring and observability category.