AI Agent Memory: What to Store, Retrieve, and Forget

“Give the agent memory” sounds like one feature. It is at least four different storage problems wearing the same label.

If you put every message into a vector database and retrieve the closest chunks, you have built a memory-shaped search system. It may recall an old preference. It may also surface a revoked instruction, another user's data, or a confident mistake the agent wrote three months ago.

Memory design starts with what the agent is allowed to remember—and why.

Separate four kinds of memory

These categories need different write policies, retrieval strategies, and deletion semantics. LangGraph's memory documentation uses a similar distinction between thread-scoped short-term memory and longer-lived semantic, episodic, and procedural memory.

Working memory belongs to the workflow

Working memory is the state required to finish the current task:

{
  "task_id": "travel_481",
  "goal": "compare three conference itineraries",
  "constraints": {
    "arrival_before": "2026-10-12T18:00:00+02:00",
    "max_budget_eur": 900
  },
  "completed_steps": ["search_flights"],
  "pending_steps": ["search_hotels", "compare_options"]
}

Store structured state outside the prompt. Reconstruct the prompt from the state needed for the next decision. This makes resumption, debugging, and approval much easier than treating the full transcript as the only database.

Long conversations require compaction, but summaries can erase constraints. Preserve critical values as typed fields and summarize the conversational material around them.

Semantic memory needs provenance

A semantic memory is a claim the system may reuse later:

{
  "subject": "user_817",
  "fact": "prefers concise Python examples",
  "source_trace_id": "trace_42",
  "confidence": 0.9,
  "created_at": "2026-08-20T12:00:00Z",
  "expires_at": null,
  "status": "active"
}

Provenance matters because facts change and models infer incorrectly. Store where the memory came from, when it was written, whether the user confirmed it, and how it can be revoked.

Do not write every model inference. “The user asked about Kubernetes” is an event. “The user is a Kubernetes administrator” is an inference. Treat them differently.

Good candidates for semantic memory are stable, useful, low-risk, and easy to correct. Sensitive attributes, speculative traits, credentials, and temporary emotions are usually bad candidates.

Episodic memory is not a raw transcript archive

Episodic memory helps the agent reuse a previous successful approach or avoid a known failure. The useful unit is often a reviewed trajectory:

Situation: deployment failed after a schema migration
Action: compared application and migration versions, then rolled forward
Outcome: service restored without data rollback
Lesson: verify migration version before restarting workers

Store episodes only when the outcome is known. A failed workflow without an explanation is a poor example. A successful workflow that violated policy is worse.

Retrieve episodes by task structure, not merely lexical similarity. The relevant experience may share the same constraints and tools without sharing the same nouns.

Procedural memory should behave like code

System instructions, policies, and operating procedures need owners, versions, tests, and deployment controls. Do not let the agent silently rewrite its own governing rules into long-term memory.

Useful procedural metadata:

• policy version;
• effective date;
• owning team;
• approval record;
• applicable tenants or regions;
• regression-test version;
• superseded policy identifier.

If a user says, “From now on, skip approval for refunds,” that is not a preference. It conflicts with procedural memory and must be rejected or routed to an authorized policy workflow.

Choose when memory writes happen

There are two common write paths.

Hot-path writes

The agent writes memory before responding. The update is immediately available but adds latency and makes every request a write-sensitive workflow.

Use this for explicit user commands such as “Remember that I prefer euros.” Validate and confirm the write.

Background writes

A separate process reviews completed traces and extracts candidate memories. This keeps the response path fast and allows stricter filtering, deduplication, and human review.

Use this for inferred preferences and episodes. A background process can compare a candidate with existing facts before creating another near-duplicate.

The split is a product decision, not a framework default.

Retrieval needs a budget and a reason

Memory retrieval competes with system instructions, user input, documents, and tool results for context.

For each memory injected into a prompt, record:

• why it was retrieved;
• its type and owner;
• relevance score;
• age and expiration;
• provenance;
• access-control decision.

Use typed filters before similarity search. Retrieve only memories belonging to the correct user and tenant. Then rank by task relevance, recency, confidence, and confirmed status.

More memory is not automatically better. Set a token or item budget and evaluate whether each memory category improves task success. The LLM observability guide provides the trace structure needed to connect retrieved memories with outcomes.

Forgetting is part of the architecture

Every memory store needs deletion and correction paths.

• explicit user deletion;
• tenant offboarding;
• retention expiry;
• superseded facts;
• policy revocation;
• low-confidence decay;
• removal of poisoned or unsafe content.

Deletion must cover the primary store, vector index, caches, summaries, and derived memories. Keeping a tombstone can prevent a deleted fact from being recreated immediately by a background extractor.

Memory also expands the prompt-injection attack surface. A malicious instruction stored today may activate during an unrelated task later. Include memory in the prompt-injection testing plan.

Evaluate memory as a component

Build cases for:

• writing a valid explicit preference;
• declining a sensitive or speculative memory;
• retrieving the correct fact among similar facts;
• not retrieving another user's memory;
• preferring a corrected fact over the original;
• ignoring an expired instruction;
• using a successful episode without copying irrelevant steps;
• deleting a fact across all derived stores;
• completing a task correctly with and without memory.

Measure task success, incorrect-memory use, cross-user leakage, stale-memory use, write precision, retrieval precision, latency, and added tokens.

Memory design checklist

• Working, semantic, episodic, and procedural memory are separate concepts.
• Critical task state is structured outside the transcript.
• Long-term facts include provenance, status, and correction paths.
• Episodes are stored only after outcomes are known.
• Procedures are versioned and cannot be rewritten by ordinary users.
• Hot-path and background writes have explicit policies.
• Retrieval enforces identity and tenant boundaries before ranking.
• Context injection has an item or token budget.
• Deletion covers indexes, caches, summaries, and derived memories.
• Memory contribution is evaluated against task success.

Sources and further reading

• LangChain memory overview
• CoALA: Cognitive Architectures for Language Agents
• LangGraph persistence concepts