It's tempting to treat an LLM's context window as its "memory" — after all, anything in the context is something the model can reference. But the context window is better understood as working attention: it's volatile, size-limited, and discarded the moment the session ends. True memory — information that persists and can be deliberately retrieved later — has to be built as a separate system around the model.

Agent memory architecture is the design of that system: what gets stored, where, in what form, and how it gets back into context when it's relevant.

Short-term vs. long-term memory

Short-term (working) memory is everything currently in the context window: the system prompt, the conversation so far, recent tool outputs. It's fast — no retrieval step needed, the model just reads it — but bounded by the context window size, and it vanishes when the session ends unless explicitly saved.

Long-term memory lives outside the context window, in a database, file store, or vector index. It survives across sessions and can hold far more than any context window, but requires a retrieval step to pull relevant pieces back into context when needed — which means the agent (or the orchestration layer) has to decide what's relevant and when.

The core engineering problem of agent memory is managing the handoff between these two: deciding what from working memory is worth persisting, and what from long-term storage is worth retrieving back into working memory for the current task.

Episodic vs. semantic memory

Within long-term memory, it's useful to separate two distinct kinds of information, borrowing a distinction from cognitive psychology:

Episodic memory — records of specific past events. "On March 3rd, the user asked to reschedule their flight and I booked seat 14C." "Last time this error occurred, restarting the worker process fixed it." Episodic memories are concrete, timestamped, and specific — they answer "what happened before."

Semantic memory — distilled, general facts abstracted away from any specific event. "The user prefers morning flights." "This error is usually caused by a stale cache." Semantic memories are the generalizations an agent (or a separate summarization process) extracts from many episodes over time.

The practical difference matters: episodic memory is cheap to write (just log the event) but expensive to use directly (you'd need to re-read and re-reason over many raw logs). Semantic memory is expensive to produce (it requires summarization or extraction) but cheap to use (a short fact slots directly into a prompt).

json

Note

Full episodic-plus-semantic memory systems with automated distillation are still mostly a research topic. Most production agents get far simpler and still get most of the benefit: log every session transcript (episodic, raw), periodically run a summarization pass over recent sessions to extract durable facts into a small structured profile (semantic, curated), and inject that profile — not the raw logs — into the system prompt of future sessions. The sophistication should match the actual need, not the state of the art.

Retrieval: getting memory back into context

Long-term memory is useless if the agent can't find the right piece of it at the right time. The two common retrieval strategies:

Similarity-based retrieval (vector search) — embed each memory, embed the current query or situation, retrieve the top-k most similar memories. Works well for "has something like this come up before" queries, the same underlying mechanism as RAG.

Structured lookup — store memory as structured fields (user_id, preference_type, value) and query it like a normal database. Works well for facts you know you'll need to look up by a specific key, like user preferences or account state, where similarity search would be overkill or imprecise.

Many production systems use both: structured lookup for known, frequently-needed facts, and vector retrieval for the long tail of "something relevant might be in here somewhere."

Memory types at a glance

Memory type	What it holds	Lifespan	Typical retrieval
Working / in-context	Current conversation, recent tool outputs	Single session	None — already in context
Episodic	Specific past events, logged as they happen	Persistent, grows over time	Vector or keyword search over logs
Semantic	Distilled general facts, extracted from episodes	Persistent, periodically updated	Direct lookup or injected into system prompt
Scratchpad / task state	Intermediate results during one multi-step task	Current task only	Passed directly between steps, not retrieved

Note

Logging everything forever sounds safe, but a long-term memory store that grows without pruning becomes slower to search, more expensive to maintain, and more likely to surface stale or contradictory facts during retrieval. Production memory systems need expiry policies, periodic consolidation (merging near-duplicate memories), and conflict resolution for when a new fact contradicts an old one.

What's next

Memory, planning, and tool use all contribute to how well an agent actually performs — but you can't know whether any of it is working without measuring it properly. The next lesson covers agent evaluation methodology: why grading a multi-step agent is a fundamentally harder problem than grading a single LLM call.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →