THE SHORT ANSWER
Measure total cost per accepted task, including model calls, context, tools, retries, verification, latency and human review. Use deterministic steps for exact work, validated smaller models for routine decisions, selected context, bounded retries, tool budgets and explicit stopping conditions. Add multiple agents only when measured gains justify their overhead.
Count the whole trajectory
| Driver | Why it grows | Control |
|---|---|---|
| Model calls | Each loop step and repair can invoke a model | Route by task and cap steps |
| Context | History, tool definitions and results accumulate | Select state and evidence |
| Reasoning | Hard cases consume more time or compute | Escalate only where needed |
| Tools | Search, APIs and code have their own costs | Use allowed sets and budgets |
| Retries | Ambiguous failures repeat work | Classify errors and bound attempts |
| Multiple agents | Context and coordination are duplicated | Require a measured specialist benefit |
| Verification | Checks and human review require resources | Scale review to consequence |
Optimise architecture before wording
- Remove model calls from deterministic rules.
- Choose the smallest validated model for each task class.
- Retrieve only relevant evidence and carry structured state.
- Limit available tools to the current step.
- Set attempt, time, tool-call and monetary budgets.
- Stop on verified completion, hard failure or escalation.
- Measure cost per accepted task after every change.
This extends the AI for Cost & Performance Optimization framework into a multi-step loop. A cheaper call can still produce a more expensive task when it increases retries or review.
Evidence & context: Microsoft Learn · OpenAI Developers
Spend capability where uncertainty earns it
Use a fixed path for known cases, a smaller model for routine classification and a more capable model for difficult categories that pass an escalation rule. Test the routing decision itself: a cheap model that sends hard cases down the wrong path can create larger downstream costs.
Keep fallbacks finite. A system that alternates models without learning from the error is a loop, not a recovery strategy.
Pair cost with quality and value
- Total and accepted tasks.
- Calls, tokens, tool use and latency per task.
- Retry, escalation and human-review rates.
- Quality and severe failures by category.
- Cost per accepted task and value tied to the original goal.
Evidence & context: Anthropic
Sources & further reading
- Building effective agents
Anthropic. A provider's engineering taxonomy of agents and workflows, not a universal industry definition. We use the conceptual distinction, not its changing product recommendations.
- Prompt caching
OpenAI Developers. Provider-specific guidance on reusing repeated prompt prefixes to reduce latency and input cost. Availability and implementation details vary by model and provider.
- Evaluate a model router
Microsoft Learn. Official guidance for evaluating routing across representative workloads using quality, cost, latency and policy criteria. It is not evidence that routing always improves results.
- Demystifying evals for AI agents
Anthropic. A provider's engineering guidance on multi-turn agent evaluation, checked 13 September 2026. Examples inform evaluation design but do not establish universal pass thresholds.
Examples and exercises are illustrative unless attributed to a source. No independent expert review is claimed.
A correction, a counterexample or an experience worth sharing?
Join the conversation ↗