Memory in AI Agents: Short-Term and Long-Term Memory Architectures

TL;DR — What makes an AI agent "intelligent" is not really the model itself, but what you remind that model of. A stateless LLM call is like a consultant who starts from scratch every single time — it knows nothing of what you discussed yesterday. If you want to build a real agent, you need to separate short-term working memory (the context window, the scratchpad, message history) from long-term memory (episodic, semantic, procedural), know when to write and when to forget, and design all of this in line with the "Agentic AI" guideline the Turkish Personal Data Protection Authority (KVKK) published in March 2026. In this piece I'm distilling a practical memory architecture guide from what I've learned in the field.

One of the misconceptions I run into most often in enterprise projects is this: "GPT-4 or Claude is already so smart, let's build an agent and it'll just learn on its own." No, it won't. Large language models operate on frozen weights once training is complete. When you make an API call, the model only sees what's in that prompt at that moment; it has no idea what you talked about in the previous call, what decision was made, or what mistake was committed. In technical literature this is called a "stateless" architecture, and it's actually a deliberate design choice — it's what allows the model to scale, run in parallel, and behave predictably.

But this is exactly where the difference between an agent and a "chatbot" shows up. A customer-service chatbot can get away with one-off, short question-and-answer exchanges. But an agent — say, a system that manages a procurement process, follows up on a customer complaint for weeks, or makes changes to a codebase in a software project — has to remain consistent over time. At a logistics company I consulted for, the agent's inability to answer "what disagreement did we have with this customer before?" in every conversation was a trust-destroying problem. When the customer said "I told you about this last week," the agent's "Sorry, I don't remember this conversation" response was an experience that damaged corporate reputation.

There are four core areas where stateless architecture falls short:

1. Continuity: In multi-step tasks, the agent has to remember each step it has taken; otherwise it loops or repeats the same mistake.
2. Personalization: An agent that doesn't remember a user's preferences, past decisions, and business context behaves like an intern starting from zero every single time.
3. Learning and improvement: If an agent cannot draw a lesson from a mistake, it repeats that mistake forever. Without procedural memory, "accumulating experience" is impossible.
4. Long-term goal tracking: In complex workflows (a supply-chain optimization, or a project that spans months), the agent must not lose sight of the "big picture."

To meet these four needs, engineers learned to add a memory layer not to the model itself, but to its surroundings. The rest of this piece is exactly about how that layer is designed.

Short-term memory resembles a human's working memory — it holds the information you're currently operating on, but its capacity is limited and it isn't permanent. In the agent world, this consists of three components:

1. The context window. The amount of tokens the model can see in a single call. Most models in use today have context windows ranging from 128K to 1M tokens, but a "bigger window" does not mean "unlimited memory." As the context window grows, the model's ability to find the most relevant information (retrieval accuracy) tends to decline — this is known in the literature as the "lost in the middle" problem. In other words, simply enlarging the window and stuffing everything into it isn't a solution; it's a new problem.

2. The scratchpad. A temporary space where the agent writes down its intermediate steps, calculations, and "thinking" process while solving a task. In agent architectures like ReAct (Reasoning + Acting), the scratchpad keeps the record of the "Thought → Action → Observation" loop. This information is usually erased once the task is finished; it is not persistent.

3. Conversation history (message history). The raw dialogue between the user and the agent. In short conversations, placing this history as-is into the context window may be sufficient. But as the conversation grows longer (say, a support call with 50-100 messages), carrying the raw history becomes both costly and performance-degrading.

This is where an important engineering decision comes in: when should summarization happen? One common approach is the "sliding window + rolling summary" technique: the last N messages are kept raw, while older messages are periodically summarized and compressed into a single "conversation summary" block. In a support agent we built for an e-commerce client, we applied this approach as follows: the last 10 messages were kept as raw text, while everything before that was kept as a structured 2-3 sentence summary such as "customer wants to return product X, reason Y, solution Z was previously offered but rejected."

The critical property of short-term memory is this: it is generally lost once the session ends. When you close an agent conversation, everything in the context window vanishes — unless, of course, you've set up a separate mechanism to make it persistent. This is exactly where long-term memory comes into play.

Long-term memory is information that persists across sessions (cross-session), stored by the agent in an external storage system (database, vector store, file system). Borrowing a classification from cognitive science, we divide it into three types: episodic, semantic, and procedural memory. This distinction isn't academic decoration — each type requires a different engineering problem and a different storage strategy.

Episodic Memory: "What Happened in the Past?"

Episodic memory is the record of specific events the agent has experienced, of past interactions. Time-stamped, context-specific records such as "On January 12th, customer X complained about product Y and we offered solution Z." This resembles the kind of memory a human uses when recalling "what did we discuss in that meeting last year."

In enterprise agent projects, episodic memory is typically critical in scenarios such as:

• A customer-representative agent remembering all previous interactions with the same customer
• A coding assistant remembering which architectural decisions were made earlier in a project
• A sales agent knowing which offers were previously discussed with a prospect

Episodic memory is generally kept as event-based records and includes metadata such as timestamp, session ID, and participants.

Semantic Memory: "What Do I Know in General?"

Semantic memory is generalized, time-independent information and facts. Information such as "this customer's company operates in sector X, has 200 employees, payment terms are net 30 days." This information may have been extracted from a specific conversation, but it is now stored as a persistent "fact" independent of that conversation.

An important subset of semantic memory is the user profile and preferences: things like "this user prefers a summary report over technical detail" or "this customer is price-sensitive and responds positively to discount offers." This kind of information is gold for personalization, but it's also — as I'll return to in the KVKK section shortly — the riskiest category from a personal-data-protection perspective.

Procedural Memory: "How Do I Do This?"

Procedural memory covers the skills, rules, workflows, and "how-to" knowledge the agent has learned. In human cognitive science it's like learning to ride a bicycle — learned once, then automated. In the agent context, this typically shows up as:

• Versions of the system prompt that are updated over time and contain learned instructions
• Workflow rules encoded as "for this type of request, follow these steps"
• "Lesson" records extracted from past mistakes and used in the future — for example, "when the user asks for format X, never use tool Y, it failed last time"

Of the three types, procedural memory is the least standardized from an engineering perspective. Some teams manage it simply as human-approved rule lists appended to the system prompt; others use automatic "reflection" loops that let the agent accumulate lessons from its own mistakes automatically. The latter approach is powerful but risky — left unsupervised, an agent can learn wrong generalizations and make them permanent.

The table below summarizes the three memory types:

So how are these three types of memory technically implemented? The most common approach is vector-based storage and retrieval. The logic goes like this:

1. The information to be stored (a conversation summary, a fact, a rule) is turned into text.
2. This text is converted into a numeric vector via an embedding model — this vector represents the semantic content of the text in a high-dimensional space.
3. The vector is stored in a vector database (Pinecone, Weaviate, Qdrant, pgvector, Chroma, etc.) along with metadata (user ID, timestamp, memory type, source).
4. When a new task arrives, the current context is also converted into a vector, and a similarity search (cosine similarity, dot product, etc.) is run against the database to fetch the top-k most relevant records.
5. The retrieved records are injected into the context window so the model can "remember" them.

This flow actually uses almost the same technical infrastructure as RAG (Retrieval-Augmented Generation) — but the purpose and content differ (I'll detail this difference shortly). A simple architecture diagram looks like this:

User Input
    │
    ▼
[Embedding Model] ──► Query Vector
    │
    ▼
[Vector Database] ──► Similarity Search (top-k)
    │
    ▼
[Relevant Memory Records]
    │
    ▼
[Injection into Context Window] ──► LLM Call ──► Response
                                          │
                                          ▼
                                [New Information Written to Memory]

One important pitfall I see in practice: teams generally prefer to lump everything into a single vector index. But keeping episodic, semantic, and procedural memory in separate indexes (or at least filterable via separate metadata tags) improves both retrieval quality and data governance — especially important for KVKK compliance. At a banking client of ours, we kept episodic memory (customer interaction history) in a separate index with a 90-day automatic deletion policy, while semantic memory (product information, general policies) was kept indefinitely but in an index that was regularly validated.

Some frameworks (LangChain's memory modules, LlamaIndex's memory classes, the file-based memory approaches used in Anthropic's Claude agent SDK, etc.) offer this pattern as ready-made components. But using a ready-made library doesn't mean you don't need to make the right architectural decisions — how long a piece of information will be stored, and what information should never be stored at all, are business and legal decisions, not engineering ones.

The most overlooked step in memory architecture is the decision of "when and what to write." A naive approach is to write every message to memory as-is — but this both bloats storage cost and degrades retrieval quality (because it becomes harder to find "the needle" among irrelevant, noisy records).

A mature memory-writing strategy generally includes these steps:

1. Filtering. Not every interaction adds value to memory. Messages like "hello" or "thanks" are usually skipped. The decision to write is often made through a separate LLM call or a rule-based filter: "Is there new information in this conversation worth storing in memory?"

2. Summarization. Instead of the raw conversation, the essence of the information is extracted and stored. A summary like "the customer asked the same question three times, and accepted solution X on the third try" is far more compact and useful for retrieval than the raw dialogue.

3. Structuring. Using as structured a format as possible (JSON, key-value pairs) instead of free text increases consistency and makes future querying easier. For example, a semantic memory record might look like this:

{
  "type": "semantic",
  "subject": "customer_123",
  "fact": "payment_terms",
  "value": "net_30_days",
  "confidence": 0.95,
  "source_session": "sess_8821",
  "last_updated": "2026-06-14"
}

4. Deduplication. Writing the same information to memory repeatedly in different phrasings means both wasted storage and an accumulation of conflicting records. Mature systems check whether a similar record already exists in memory before writing a new one (again via vector similarity) and update it if it does, or create a new record if it doesn't — this is called "upsert" logic.

5. Consolidation. Periodically transforming a large number of accumulated episodic records into higher-level semantic knowledge over time. For example, individual episodic records like "customer complained about delivery delays 3 times in the last 6 months" can turn, during consolidation, into a single semantic profile fact like "this customer is sensitive to delivery delays and prefers proactive communication." This resembles the memory consolidation that happens in human memory during sleep — I like this biological metaphor because it explains very clearly to engineers why this step is necessary.

At a manufacturing company where we built a maintenance-support agent, we ran consolidation as a weekly batch job: the episodic records accumulated that week were fed to an LLM and asked "what general patterns can be extracted from these records?" and the resulting suggestions were added to semantic memory only after human approval. This human-approval step is critical — if you leave automatic consolidation completely unsupervised, the model can produce wrong generalizations (unfair and inaccurate inferences like "this customer always complains").

Storing the right information isn't enough; retrieving the right information at the right moment is at least as hard a problem. Several factors affect retrieval quality:

Semantic similarity alone isn't enough. Retrieval based purely on vector similarity often surfaces records that are superficially similar but contextually irrelevant. That's why practical systems generally use hybrid retrieval: vector similarity + keyword search (like BM25) + metadata filtering (time range, user ID, memory type) working together.

Re-ranking. Candidate records retrieved in the first pass (say, top-50) are re-ranked more precisely with a second model, and the most relevant k (say, top-5) are added to the context window. This two-stage approach provides a good balance of speed and accuracy.

Recency weighting. Especially in episodic memory, more recent events are generally more relevant. Adding a "recency" factor to the retrieval score (for example, via exponential decay) prevents old, outdated information from unnecessarily surfacing.

Context-triggered retrieval. Instead of querying memory at every step, identifying the moments when the agent genuinely needs memory. For example, when a user says "like we discussed last time," this is an explicit trigger. But often the trigger is implicit, and the agent itself needs to evaluate the question "do I need past information to complete this task?"

Personally, the biggest mistake I see in the field is designing retrieval with a "always fetch as much information as possible" mindset. This leads to what's called context bloat — I'll detail this in the pitfalls section shortly. The right approach is to retrieve "little but relevant" information.

Human memory has an ability at least as important as learning: forgetting. Without deliberate forgetting mechanisms in agent memory, the system becomes inefficient over time, costs rise, and — more importantly — stale information starts corrupting correct decisions.

Practical forgetting strategies include:

• TTL (time-to-live) policies: Assigning an expiration date to every memory record. Episodic records typically get shorter TTLs (30-90 days), while semantic records may have longer or indefinite TTLs.
• Access-frequency-based pruning (LRU logic): Archiving or deleting records that haven't been accessed for a long time.
• Salience scoring: Keeping a score for how "important" each record is, and pruning low-importance records first. This score can be computed from signals like how many times the record was retrieved, how recently it was used, or whether a human flagged it as "important."
• Compression: Instead of deleting, turning old records into a more compact summary and storing that — as part of the consolidation process.

From a cost-control perspective, the memory system actually generates three separate cost categories: (1) embedding generation cost, (2) vector database storage and query cost, and (3) LLM call cost for the extra tokens added to the context window. In a large-scale customer-service agent, uncontrolled memory growth can start carrying thousands of unnecessary tokens per call at some point — this seriously affects both response time and the bill. That's why forgetting isn't just "housekeeping" — it's directly a cost-engineering concern.

This is one of the most common points of confusion in my training sessions: "Isn't RAG memory already?" Partly yes, but not exactly. It's worth clarifying the difference:

In practice, these two systems can share the same technical infrastructure (embedding + vector store + retrieval), and in some architectures they can even live as different collections in the same vector database. But they need to be kept conceptually separate because their governance rules differ: if a document in RAG is wrong, you fix the source and re-index it; if a record in agent memory is wrong (say, an incorrect inference about a customer), this can directly corrupt every future interaction with that user and often touches on personal data protection issues.

Some modern architectures combine the two into a "hybrid memory": the RAG layer provides general knowledge while the agent memory layer provides personalized context, and the two are merged in the same prompt. This is the approach I recommend — designing them as layers that complement, rather than replace, one another.

The most concrete business value of long-term memory shows up in multi-session persistence and personalization. When a user talks to an agent today and comes back tomorrow, the agent "not starting from scratch" — that's the feature my enterprise clients request most.

In practical implementation, this is usually structured as follows:

• A persistent user profile (semantic memory) is maintained for each user: preferences, past decisions, communication style.
• At the end of each session, any important information extracted from that session (if any) is written to the profile or to episodic history.
• When a new session starts, the agent first retrieves the user profile and the last N episodic records and adds them to its context — this is sometimes called "memory priming."

The power of personalization is undeniable — at an education-consulting platform, the agent remembering which topics the user struggled with in previous sessions and adapting new content accordingly noticeably increased user satisfaction. But balance needs to be struck here: excessive personalization risks putting the user in a "filter bubble," and more importantly, the more personal data you accumulate, the more legal and ethical responsibility you take on. My recommendation for striking this balance: keep the minimum data needed for personalization, and don't accumulate data on a "might need it later" basis.

How do you know a memory system is "working well"? This is a step most teams skip, yet it's one of the most critical. Evaluation is generally done across three dimensions:

1. Accuracy: Does the information written to memory reflect reality? Hallucination risk exists here too — the model can draw a wrong inference from a conversation and write it to memory as "fact."

2. Retrieval quality (precision/recall): Can the needed information be retrieved at the right time (recall), and is the retrieved information actually relevant (precision)? To measure this, teams typically use labeled test sets: gold-standard scenarios prepared as "for this query, these records are expected to be retrieved."

3. Downstream task performance: Ultimately what matters is whether memory improves the agent's actual task success. Through an A/B test setup, task completion rate, user satisfaction, and error rate can be compared with memory active versus inactive.

There should also be a forgetting evaluation: testing whether the system correctly ages out information that is no longer valid (staleness detection). At one client, the agent still remembering a subscription that had been canceled 8 months earlier as "active" led to a serious loss of trust — a classic example of retrieval working fine while the forgetting/update mechanism was broken.

Here are the memory architecture mistakes I see most often in the field:

"

Context bloat: Stuffing as much historical information as possible into the context window on a "just in case" basis. Result: the model both slows down and becomes more expensive, and due to the "lost in the middle" effect it misses the actually important information.

"

Stale memory: Information gets written once and is never updated or verified again. The customer's address changed, but the agent still "remembers" the old one.

"

Lack of deduplication: Writing the same information repeatedly at different times in different forms leads to an accumulation of conflicting records and degraded retrieval quality.

"

Privacy/security leakage: The most serious risk. Information in one user's memory leaking to another user due to a flawed prompt design or a retrieval error. In multi-tenant systems, memory isolation is one of the most critical security requirements of architectural design.

"

Leaving automatic consolidation unsupervised: Wrong generalizations the model "learns" on its own being written into permanent memory without human approval. Over time this can lead the agent to make biased or incorrect decisions.

"

Not testing memory: Failing to systematically test retrieval quality, forgetting behavior, and privacy isolation before going to production — most teams postpone this as "we'll deal with it later," and the problem then surfaces as a customer complaint in the live environment.

In every agent project operating in Turkey, or serving Turkish users, memory architecture design is no longer just an engineering decision — it's also a compliance decision. The Personal Data Protection Authority (KVKK) published a guideline titled "Etken Yapay Zeka" (Agentic AI) in March 2026, specifically drawing attention to the personal-data-processing risks of AI systems that make autonomous decisions and take actions on their own. What makes this guideline significant is this: unlike traditional AI systems, agentic systems don't just process personal data — they accumulate it over time, correlate it, and use that accumulated data to make autonomous decisions. That's exactly the memory architecture described in this piece.

At the organizations I've advised, here are the concrete points we now pay attention to in memory architecture design following this guideline:

1. Every piece of personal data written into agent memory is a processing activity. The moment a customer's name, contact information, purchasing habits, health status, or financial information is written into episodic or semantic memory, the general obligations under Law No. 6698 (legal basis, notification/transparency, purpose limitation, data minimization) kick in. The defense of "the agent wrote it automatically, we didn't enter it manually" does not eliminate legal responsibility.

2. Data minimization should sit at the center of memory design. Consolidation and summarization steps should be used not just for performance, but also to prevent unnecessary personal data from leaking into permanent memory. For example, a health detail mentioned in a customer-support conversation should not be written into semantic memory as a "permanent fact" unless it is genuinely necessary for the business purpose.

3. Anonymization and pseudonymization. Wherever possible, using internal reference IDs (like customer_id) instead of directly identifying information in episodic records reduces the risk of direct personal-data leakage even if the memory system is breached.

4. KVKK Article 15 and the expectation of human oversight. KVKK's approach — which anticipates human intervention and oversight in automated/autonomous decision-making processes — is especially important for autonomous decisions fed by agent memory. If an agent reaches an automatic conclusion about a customer based on information in its memory (for example, "this customer is risky, restrict service"), that decision is expected to go through human approval, or at least be contestable/auditable. In practice, this means establishing a "human-in-the-loop" gate for memory-driven autonomous decisions — especially for decisions that produce legal consequences or significantly affect a person.

5. The right to erasure and correction needs a technical counterpart. Under KVKK, data subjects have the right to request deletion (the right to be forgotten) and correction. In your memory architecture, can you technically execute "delete all episodic and semantic records for this user"? Is this level of granular deletion possible in your vector database? Asking this question during the architecture design phase is far cheaper than asking it after going to production.

My observation in the field is this: since the Agentic AI guideline was published, most of my enterprise clients now request a "memory data governance plan" at the very start of an agent project, before architecture design even begins. This is a positive development — because retrofitting memory architecture for compliance afterward is far more costly than designing it correctly from the start.

Bringing together the concepts covered in this piece, here is a concrete sequence I recommend when starting an agent project:

1. Do a needs analysis. Does your agent actually need long-term memory, or is a well-designed short-term working memory sufficient? Not every agent requires long-term memory — for simple, one-off tasks, this can be over-engineering.
2. Separate the memory types. Design episodic, semantic, and procedural memory as separate from the start; separating them later is much harder.
3. Define the writing policy. What gets written, when it gets summarized, who (a human or the model) approves it — put this down in a document before writing any code.
4. Design the forgetting policy on day one. TTLs, archiving rules, and deletion mechanisms should not be a feature added later — they should be part of the core architecture.
5. Build data governance together with KVKK compliance. Especially after the Agentic AI guideline, having legal and engineering teams jointly review memory design is no longer a luxury — it's a necessity.
6. Measure it. Test retrieval quality, forgetting behavior, and privacy isolation regularly, both before and after going to production.

Memory is the least glamorous but most decisive layer of an agent project. No matter how powerful the model itself is, an agent that doesn't remember the right information at the right time remains untrustworthy in the eyes of its users. In the enterprise projects I've seen, the most successful agents were not the ones using the biggest model — they were the ones whose memory was designed with the most care.