Memory Architectures for AI Agents: Designing Short- and Long-Term Memory

TL;DR — For AI agents, memory is a design problem as important as "how smart the model is." We split it in two: short-term memory that lives inside the context window (system prompt, chat history, recent tool calls, chain-of-thought) and long-term memory that persists across sessions (user preferences, task history, learned knowledge). Long-term memory itself divides into semantic (facts), procedural (workflows) and episodic (past events). This post walks through how to take both to production: vector databases, summarization buffers, memory scoring/decay policies, the ACE (Agentic Context Engineering) loop, failure modes like "context rot" and memory poisoning, and the obligations that storing user data triggers under Turkey's KVKK — with a concrete reference architecture and checklist.

The disappointment you feel the first time you ship an agent is usually not about the model's intelligence. The model understands the question, calls the right tool, writes a nice answer. But by the third message it forgets what the user said two turns ago; or when the same user returns tomorrow, it treats them like a stranger. This is the most common problem I see in the field: the agent's "brain" is good, but its "memory" is missing or wrongly designed. I wrote this to put on paper the whiteboard diagram I've drawn dozens of times for the teams I consult.

Large language models are stateless. On every API call, whatever you send is "the entire world" for that moment; once the call ends, the model remembers nothing. In chat apps we hide this fact by resending the history over and over. But if an agent runs a task lasting weeks, makes hundreds of tool calls, and works with the same user for months, "resend everything" becomes both impossible and expensive.

Memory engineering fills exactly this gap: the art of deciding which information must enter the context window, which stays outside in a persistent store to be recalled on demand, and which is forgotten entirely. This is a different skill from prompt writing; it resembles designing an operating system's virtual memory manager. You continuously move data between a small, expensive "RAM" (the context window) and a large, cheap "disk" (the persistent store).

Internalize this early: enlarging the context window does not solve memory. Even 200K or million-token windows can't rescue the "stuff everything in" strategy, because of the "context rot" phenomenon I'll describe below. Memory is not a capacity problem, it's a selectivity problem.

Short-term memory is everything you send to the model in the current API call. We can break it into parts:

• System message: The agent's identity, task, rules, tone. Usually long and static.
• Working memory: The current state of the task at hand — what step we're on, what intermediate results we have, what the user approved.
• Chat history: The sequence of human and assistant messages.
• Recent tool calls and results: Functions the agent just invoked and the data they returned.
• Chain-of-thought: The model's reasoning traces and plan steps.

All these components share the same limited token budget. As a task grows, chat history and tool outputs balloon; at some point they no longer fit. Two classic strategies come in here: truncation and summarization.

Truncation or summarization?

Truncation is the simplest method: drop the oldest messages, keep the last N. Sliding-window logic. Fast, predictable, cheap. But brutal — it can throw away a critical constraint the user gave two hours ago (e.g., "budget must not exceed 50,000 TL").

Summarization is more elegant: you have the model summarize old turns and replace them with a short summary, freeing space while preserving information. But there's a trap I see often in the field: over-summarization. If you re-summarize a long dialogue every time, you take the summary of the summary of the summary, and each turn a bit more information evaporates. A subtle preference the user expressed at message 40 has dissolved in the summary chain by message 60. The agent suddenly faces "why did you forget that?"

In practice I recommend a hybrid approach:

"

Keep the last K turns verbatim, compress everything older into a rolling summary; but store the critical "facts" (user constraints, decisions, identity) separately in a structured "working memory" block, outside the summary.

This way you pull the information that summarization loss would hurt most out of the summary stream and protect it. Think of the working-memory block as a JSON: user name, language preference, active goal, approved budget, finalized decisions. You rewrite this block deterministically every turn; you don't beg the model to "not forget."

Context rot: why does a growing context degrade quality?

Our intuition says "more context = better answer." Reality is more complex. As the context window fills, the model's attention scatters; irrelevant or repeated information overshadows the relevant signal. The industry calls this context rot: as context grows — especially as noise, stale tool outputs, and repeated summaries accumulate — output quality gradually degrades. The model can technically "see" 200K tokens, but it can't attend to all of them equally and accurately.

The practical consequence: don't use the context window as a trash can. Throwing in every tool output, every intermediate step, every raw log line is easy in the short term but slowly blinds the agent. Managing context with curation — deliberately deciding what enters and what leaves — is the most important, least discussed job of production agents.

Short-term memory disappears when the task ends or the window is cleared. Long-term memory persists across sessions, days, months. Borrowing a taxonomy from human cognition, we split it into three subtypes:

1. Semantic memory — facts and reusable knowledge

Semantic memory is context-independent facts: "This user is the CFO of an e-commerce company," "The company's fiscal year ends December 31," "The return policy is 14 days." Not tied to time or event; general truths and reusable reasoning patterns. In a customer support agent, semantic memory is product catalog and policy knowledge. It usually lives in a vector database or structured knowledge base and is injected into context via retrieval on demand.

2. Procedural memory — workflows and strategies

Procedural memory is "how to do it" knowledge: the steps to process a refund, the order in which to compile a report, the strategy to follow when hitting a certain error type. The equivalent of "muscle memory" in humans. When the agent completes a task successfully, it can write the steps it followed into a "playbook" and reuse it on similar future tasks. Procedural memory is often embedded in the system message or tool descriptions; in some architectures it's kept as a separate "strategy store."

3. Episodic memory — detailed context of past events

Episodic memory is the recollection of specific, dated events: "Last Tuesday this user reported problem X, we suggested solution Y, it didn't work." Rich, contextual, event-specific. This is the heart of personalization and the sense of continuity. When a user says "that thing we discussed last time," episodic memory is what lets the agent recall it. In practice you write each important interaction to a vector store with a timestamp and recall relevant memories by semantic similarity.

The table below summarizes these distinctions:

Memory type	What it stores	Lifespan	Typical implementation
Short-term / working	Current task state, recent turns	Session / window	Context window, summary buffer
Semantic	Facts, policies, general knowledge	Persistent	Vector DB, knowledge base
Procedural	Workflows, strategies, playbook	Persistent	System message, strategy store
Episodic	Dated events, past interactions	Persistent	Vector DB (timestamped)

Now to turning these concepts into real systems. A few concrete patterns work in the field.

Episodic and semantic recall with a vector database

The most common pattern: turn important information into embedding vectors and store them in a vector store (like pgvector, Qdrant, Weaviate, Pinecone). When a new user message arrives, you embed it, find the K most similar memories in the store, and inject them into context. This is classic RAG (retrieval-augmented generation) applied to memory.

Three things deserve attention. First, what you write: storing every message raw fills your store with noise. It's usually better to write a "worth remembering" summary extracted from the interaction ("User prefers communicating in Turkish and likes technical depth"). Second, how you retrieve: pure semantic similarity isn't enough; a hybrid ranking that combines recency, importance, and relevance often works better. Third, when you retrieve: not every turn, but at the moments the agent decides "I need to remember something" — that's both cheaper and cleaner.

Summarization buffers

For short-term memory, the rolling summary buffer is a classic and effective pattern. You keep chat history in two layers: a hot layer (last K turns, verbatim) and a cold layer (older ones, in a single rolling summary). As a new turn arrives, the oldest raw turn merges into the summary. The critical point is keeping summary generation deterministic and templated: give the model clear instructions to "update the previous summary, preserve these fields" so over-summarization loss is minimized.

Memory scoring and decay policies

Human memory forgets, and that's not a flaw but a feature. In agent memory too, keeping every memory forever is both expensive and dangerous (see stale memory and privacy below). So you assign each memory a score, and that score decays over time. A simple formula:

"

score = w1 * relevance + w2 * recency + w3 * importance − w4 * age

Memories whose score falls below a threshold are archived or deleted. Important memories (ones the user explicitly said "remember this," or ones accessed repeatedly) are boosted; memories used once and never touched again fade. This supports both token cost and KVKK's data-minimization principle at the same time.

One of the approaches standing out in 2026 is treating context not as a static prompt but as a continuously evolving "playbook." Agentic Context Engineering (ACE) improves context through a three-agent loop:

1. Generator: Tries to solve the task and produces a trajectory — the steps it took, the tools it called, the results it got.
2. Reflector: Examines that trajectory, detects errors, identifies missing context. It says, "if you had known this here, you'd have acted differently."
3. Curator: Extracts learnings from the Reflector's findings and writes them into a persistent context playbook. On the next task, the Generator starts with this updated playbook.

The beauty of this loop is that it turns memory from mere "data storage" into "learning from experience." The agent draws lessons from its mistakes, and those lessons are written permanently into procedural memory. When explaining this to teams I use an analogy: the Generator is the player on the field, the Reflector is the coach doing post-match analysis, and the Curator is the manager updating the tactics book. The player doesn't start from scratch each match; the book accumulates.

In the field and the literature I clearly see these directions:

• Hierarchical memory: Hot/warm/cold layers. The most frequently accessed information stays in context, medium-frequency in a fast cache, the rest in the vector store. Very much like an operating system's memory hierarchy.
• Multi-agent shared memory: Multiple agents writing to and reading from a shared memory layer. A research agent's findings used by a writing agent. Here consistency and concurrency bring new challenges.
• Retrieval-backed long-term memory: The vector store is now standard. The shift is from pure similarity to hybrid and graph-based retrieval.
• Memory writing/forgetting policies: What gets written and what gets forgotten is now an explicitly designed policy. The "store everything" era is ending.

Memory systems have their own characteristic ways of breaking. Knowing these in advance keeps you from waking at 3 a.m. in production.

Memory poisoning

If the agent writes wrong or malicious information into its memory, that information contaminates all subsequent decisions. Worse, an attacker can deliberately inject false "facts" into the agent's memory through user messages (the persistent form of indirect prompt injection). If the agent writes this into semantic memory, the poison becomes permanent. Defense: a validation layer before writing to memory, tagging information from untrusted sources as a "claim," and human approval for critical memories.

Stale memory

The world changes but memory doesn't. The "I currently live in Istanbul" a user wrote six months ago may be wrong today. The agent trusts stale information and personalizes wrongly. Defense: add timestamps and last-verified dates to memories, decay policies, and — when a contradiction is detected — prefer the newer and invalidate the older.

Privacy leakage

The most insidious failure. The agent may accidentally leak information held in one user's memory to another user or the wrong context. In multi-agent shared memory this risk multiplies. Defense: strictly isolate memories per user/tenant, apply access control to retrieval queries, and mask sensitive fields.

The moment you write user data into the agent's memory, that memory becomes a personal data processing activity and all obligations of KVKK (Law No. 6698) kick in. Thinking about this in pre-production architecture is far cheaper than patching it later.

• Data minimization: The urge to write everything into agent memory "just in case" directly violates KVKK's proportionality and purpose-limitation principles. Store only what's necessary for the stated purpose. Scoring and decay policies are a technical compliance tool here.
• Retention periods: Define a retention period for each memory type. Episodic memories shouldn't live forever. Set up automatic deletion/archival jobs.
• Right to erasure: When a user says "delete my data," it must cover the embedding vectors in the vector store, the summary buffers, and personal references in persistent playbooks. In practice this is hard because personal data is scattered across summaries and embeddings. Design memory records tagged with user identity so bulk deletion is possible.
• Disclosure and explicit consent: Transparently inform users that the agent remembers information about them and how it uses it.
• Cross-border transfer: If the vector store or LLM provider is abroad, KVKK's transfer regime and, where applicable, the EU AI Act's transparency obligations come into play. In high-risk uses, the AI Act's record-keeping and human-oversight requirements also concern memory.

In short: memory is not just a performance feature, it's also a compliance surface. Invite legal and security teams to the table early as you draw the architecture.

The layered reference architecture I recommend in the field looks like this:

1. Orchestration layer: A "memory manager" that decides each turn which memories enter context. It works like an operating system's memory manager.
2. Working-memory block: Structured state rewritten deterministically (user constraints, active goal, decisions). Protected from summarization loss.
3. Summary buffer: Two layers, hot (raw recent turns) + cold (rolling summary).
4. Long-term store: Vector DB (episodic + semantic) and strategy store (procedural). Isolated per user/tenant.
5. Memory write service: The gate managing what, when, and with what validation is written. The first line of defense against poisoning.
6. Retrieval service: Hybrid scoring (relevance + recency + importance), access control, masking.
7. Maintenance jobs: Decay, archival, KVKK deletion requests, consistency checks.
8. Observability: Logs tracking which memory influenced which decision — for both debugging and audit.

Let's make it concrete. Consider a support agent we built for a B2B SaaS company. Different users at the same customer firm talk to the agent over months. In the first version there was no long-term memory; every conversation started from scratch. Users complained most about "I explain the same thing from the beginning every time." When adding memory, we built these layers in sequence.

First we added the working-memory block: the active ticket number, product version, previously tried solutions. This block was rewritten every turn and kept outside the summary stream, so even in the middle of a long conversation the agent didn't re-ask "which version are you on?" Then we added episodic memory: a summary of each closed ticket (problem, root cause, solution, outcome) was written to the vector store with a timestamp. When a new ticket opened, similar past tickets were recalled and the agent could say "this firm had a similar issue three months ago, and back then this worked."

We wrote the reusable resolution steps learned from root causes into procedural memory, a strategy store. Over time the agent accumulated the most effective diagnostic order for specific error signatures on its own. We didn't build an ACE-like loop here, but we mimicked the Reflector logic with a weekly human review: an engineer examined failed conversations and added corrections to the playbook. This was a safe intermediate step before moving to a fully automatic loop, and I recommend it to you too — start automatic memory writing behind a human-supervised gate.

The result was a clear improvement in resolution time and a visible drop in repeated questions. But the real lesson was this: what created the value wasn't a model change, it was the right layering of memory. With the same model, by designing memory correctly, the agent looked far more competent.

Memory is already hard in single-agent systems; the difficulty multiplies when several agents share the same memory. A planning agent reads what a research agent wrote, and an execution agent updates it. Here classic distributed-systems problems return: race conditions, inconsistent reads, conflicting writes.

A few principles work in the field. Try to keep memory single-writer: each memory piece has a clear "owner," others only read. Make writes idempotent and timestamped so that in a conflict it's determined which one wins. And model shared memory as "events" as much as possible — an accumulating event log rather than an overwritten state makes conflicts easier to resolve. If these principles sound familiar from microservice architecture, don't be surprised; agent memory increasingly resembles a distributed state-management problem, and it's wise to draw on the accumulated knowledge in that field.

Don't see memory merely as a quality feature; it's a direct cost item. Every token entering context is money. If an agent needlessly carries tens of thousands of tokens of history every turn, your bill grows in proportion not to the task's complexity but to your memory's wastefulness. A well-designed memory — context managed with curation, smart summarization, retrieval on demand — both raises quality and lowers the bill. These two rarely conflict here; they usually improve together. In the next post, when I cover LLM cost optimization in detail, I'll touch on why prompt caching only works when there are stable prefixes; your memory design either builds or breaks that stability. If you keep your system message and procedural memory fixed in the prefix and put the variable parts at the end, your cache hit rate — and therefore your cost — improves markedly.

Before taking an agent memory system to production, go through this list with your team:

• Did we clearly separate short- and long-term memory?
• Are critical facts (user constraints, decisions) in a protected working-memory block, outside the summarization stream?
• Is summarization templated and deterministic against over-summarization?
• Are we filling the context window with noise, or managing it with curation? (context rot check)
• Did we classify long-term memory as semantic / procedural / episodic?
• Does vector-store retrieval ranking go beyond pure similarity to hybrid scoring?
• Does each memory have a score and a decay policy?
• Do we validate/tag before writing to memory? (poisoning defense)
• Do memories carry timestamps and freshness tracking? (stale-memory defense)
• Are memories strictly isolated per user/tenant? (privacy-leak defense)
• KVKK: are data minimization, retention periods, and right to erasure technically enforceable?
• Did we assess cross-border transfer and EU AI Act transparency obligations?
• Is there observability showing which memory influenced which decision?

Don't try to build everything at once. The sequence that works in the field: first, harden short-term memory — a good summary buffer and a protected working-memory block solve eighty percent of any agent's "forgetfulness" complaints. In the second phase, add episodic memory: summarize important interactions, write them to a vector store, recall relevant memories. The sense of personalization kicks in here. In the third phase, mature semantic and procedural memory; with an ACE-like loop, let the agent accumulate its own playbook. At every phase, don't leave KVKK and security controls for later; build memory isolated, scorable, and deletable so that as you scale, compliance is a ready capability rather than a crisis. Memory is the key not to your agent getting smarter over time, but to it becoming trustworthy over time; and trustworthiness is worth more than intelligence in production.