THE SHORT ANSWER
Evaluate the full system on representative tasks. Grade the final environment outcome, accepted output, tool calls, policy compliance, intervention, cost and latency. Run repeated trials where model variation matters, inspect failures by category, and combine deterministic, model-based and human evaluation according to the task.
Define task, trial, trace and outcome
A task is a test case with inputs and success criteria. A trial is one attempt. The trace records decisions, tool calls and observations. The outcome is the final state in the environment. These distinctions prevent a confident final message from passing when no action actually succeeded.
Because model behaviour can vary, repeated trials can reveal instability that one demonstration hides. Do not publish one universal pass threshold; set requirements from the consequence and operating context.
Evidence & context: Anthropic
Evaluate several dimensions together
| Dimension | Example measure | Why it matters |
|---|---|---|
| Outcome | Verified task success | Tests the actual final state |
| Quality | Accepted without correction | Reflects usability |
| Tools | Successful calls and valid arguments | Locates interface failures |
| Control | Policy violations and denied actions | Tests permission boundaries |
| Human role | Intervention and escalation rate | Shows operational burden |
| Efficiency | Cost and latency per accepted task | Connects reliability to resources |
| Value | User or business outcome | Tests whether the system is worth operating |
Match the grader to the claim
Use code for exact schemas, calculations, permissions and environment state. Use a calibrated rubric when quality is contextual. Use qualified people for domain judgment and high-impact outcomes. Model-based graders can scale nuanced checks, but they also vary and should be calibrated against human decisions.
Grade both outcome and trajectory. An accepted result reached through a forbidden tool call is not a safe success; a long harmless path may still signal cost or stopping problems.
Build a living evaluation suite
- Normal cases drawn from the intended workload.
- Rare cases with serious consequences.
- Missing, stale and conflicting information.
- Tool timeouts, partial results and permission denial.
- Attempts to redirect the agent through untrusted content.
- Budget, iteration and human-escalation boundaries.
- Regression cases from real incidents and corrections.
Evidence & context: Anthropic
Sources & further reading
- Demystifying evals for AI agents
Anthropic. A provider's engineering guidance on multi-turn agent evaluation, checked 13 September 2026. Examples inform evaluation design but do not establish universal pass thresholds.
- Building effective agents
Anthropic. A provider's engineering taxonomy of agents and workflows, not a universal industry definition. We use the conceptual distinction, not its changing product recommendations.
- Generative Artificial Intelligence Profile (NIST AI 600-1)
NIST. Risk-management guidance, including confabulation. It does not establish a universal error rate.
Examples and exercises are illustrative unless attributed to a source. No independent expert review is claimed.
A correction, a counterexample or an experience worth sharing?
Join the conversation ↗