THE SHORT ANSWER
Context is information available to the model now. Conversation history is one possible part of it. Working state records progress in the current task. Persistent memory carries selected information to later tasks. External systems remain authoritative sources. Keeping these separate reduces stale assumptions, privacy exposure and unnecessary token use.
Give each information layer a job
| Layer | Purpose | Typical lifetime | Primary risk |
|---|---|---|---|
| Context | Support the current model decision | One call or selected turns | Irrelevance and token cost |
| Conversation history | Preserve useful interaction detail | Current session | Old instructions and growing noise |
| Working state | Track steps, approvals and results | Current task | False or partial completion |
| Persistent memory | Reuse deliberately saved facts or preferences | Across tasks | Staleness, privacy and wrong identity |
| External data | Remain the source of record | Defined by the business system | Access, freshness and provenance |
Represent progress outside the prose
A reliable workflow should not depend on the model remembering that an invoice was approved somewhere in a transcript. Store structured task state such as item identity, current stage, approval status, tool result and timestamp. The controller decides which fields enter the next context.
State transitions should distinguish proposed, executed and verified. This prevents a generated plan from being mistaken for a completed external action.
Save selectively and retrieve skeptically
Persistent memory should have a defined reason, provenance, scope and deletion path. A user preference may be useful later; an inferred personal fact may be wrong or inappropriate to retain. Retrieve only memory relevant to the current goal and let authoritative current data override it.
Retrieval can select passages from an external collection, but retrieved text is context, not guaranteed truth or instruction. Preserve source identity and freshness.
Evidence & context: Lewis and colleagues, 2020 · NIST
Treat context as a controlled budget
Long histories, repeated tool descriptions and duplicated documents increase processing and can hide the decisive evidence. Context Windows and AI Cost explains how to select a working set. Provider caching can reduce repeated-input cost in supported cases, but it does not make unnecessary content relevant.
Evidence & context: OpenAI Developers
Sources & further reading
- Building effective agents
Anthropic. A provider's engineering taxonomy of agents and workflows, not a universal industry definition. We use the conceptual distinction, not its changing product recommendations.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis and colleagues, 2020. Research combining generation with retrieved material. Retrieval should not be interpreted as proof of factual accuracy.
- Prompt caching
OpenAI Developers. Provider-specific guidance on reusing repeated prompt prefixes to reduce latency and input cost. Availability and implementation details vary by model and provider.
- Generative Artificial Intelligence Profile (NIST AI 600-1)
NIST. Risk-management guidance, including confabulation. It does not establish a universal error rate.
Examples and exercises are illustrative unless attributed to a source. No independent expert review is claimed.
A correction, a counterexample or an experience worth sharing?
Join the conversation ↗