Agent Memory Architecture Tradeoffs We Learned the Hard Way

Memory sounds like a solved problem until you build an agent that needs to remember a conversation that started three sessions ago, reference a decision made mid-conversation twenty turns back, and not confuse one customer's details with another's. We burned through three architectures before landing on something stable.

The first attempt was the obvious one: dump the full conversation history into context every turn. It worked fine in the demo and fell apart within two weeks of real usage, because long-running support threads pushed context length past the point where the model reliably attended to anything from early in the conversation — it would contradict a decision made ten turns earlier because that decision had effectively scrolled out of relevance.

Attempt two: naive vector retrieval over past turns

We switched to embedding every past turn and retrieving the most semantically relevant ones each time. This fixed the length problem but introduced a worse one: semantic similarity doesn't track conversational relevance. A customer mentioning their account was 'closed' two weeks ago and asking about billing now would retrieve unrelated turns that happened to share vocabulary, while missing the actually relevant turn where they'd explained why they closed it.

Semantic similarity finds what sounds related. Conversations need what's actually related, and those aren't the same thing.

What finally worked

• A structured summary that gets explicitly updated after each turn, not regenerated from scratch — key facts, open issues, decisions made, written as discrete fields rather than prose
• Full recent turns (last 4-6) kept verbatim in context for conversational coherence
• Vector retrieval kept only as a fallback for explicitly referencing something from much earlier, not as the primary memory mechanism
• An explicit 'what changed this turn' step that updates the structured summary, so stale facts get overwritten rather than accumulating

The structured summary approach cost more engineering effort upfront than either of the first two attempts, but it's the only one that held up past a few hundred real conversations. The lesson that generalizes: memory architecture problems rarely show up in a demo with ten turns. They show up specifically at the length and messiness of real usage, so testing memory design against anything shorter than that is testing the wrong thing.