Episodic vs semantic memory in agents (and why your agent needs both)

The most common question we get from technical readers about agent memory is whether to "use a vector database" or "use a knowledge graph." It is the wrong question. The right question is what kinds of memory you need, and the answer is at least two: episodic and semantic. They are different shapes, they answer different queries, and an agent that only has one feels broken in different ways depending on which one is missing.

This post walks through the two memory categories, why they need to coexist, what each one is good at, and how they get implemented under the hood. It is the technical companion to the earlier post on agent memory.

Episodic memory: the timeline

Episodic memory is a sequence of events with timestamps. Each entry is a discrete observation: "user said X at time T," "user did Y at time T," "tool call Z fired at time T with result R." The store is a timeline; queries against it are temporal.

Useful queries against episodic memory:

• "What did I tell you about the H1 retro?" returns the thread of comments tagged with that topic, in chronological order.
• "When did I last see Mrs. Henderson?" returns the most recent visit timestamp.
• "What did Sara say in our meeting last Tuesday?" returns the relevant transcript lines.

Implementation-wise, episodic memory is almost always a database table with a timestamp index, sometimes augmented with vector embeddings for semantic search across the timeline. The retrieval layer at runtime queries by user ID + time window + topic.

The shape is right for "what happened." It is wrong for "what is generally true."

Semantic memory: the facts

Semantic memory holds stable facts about the user, the world, and the user's preferences. "Your wife is allergic to peanuts." "You prefer morning meetings to afternoon ones." "Felipe's haircut is your default Saturday booking." These facts are not bound to a specific moment; they describe a steady-state truth that the agent should know without having to look it up in the timeline.

Useful queries against semantic memory:

• "Suggest a restaurant for dinner with my wife" pulls "wife is allergic to peanuts" without searching the timeline.
• "Book me a haircut" pulls "default haircut is Felipe's, Saturday morning."
• "Schedule a meeting with Sara" pulls "Sara prefers morning, time zone Eastern."

Implementation-wise, semantic memory is often a key-value or attribute-value store keyed to entities (user, customer, preference category). It can be queried directly by attribute, without temporal scoping.

The shape is right for "what is true." It is wrong for "what happened on Tuesday at 3pm."

Why an agent needs both

Three reasons.

Different time scales. Episodic events happen at human timescales (a conversation, a meeting, a transaction). Semantic facts persist across many such events. An agent that operates at one timescale only is going to be either short-sighted (no semantic) or detail-blind (no episodic).

Different retrieval patterns. A timeline is best queried with "show me the last N events about X." A semantic store is best queried with "what is true about Y." Mixing the two retrieval patterns into a single store leads to a bad query model and slow retrieval as data grows.

Different decay rates. Episodic events naturally decay (most are operational and stop mattering after a week). Semantic facts persist (your wife's peanut allergy stays true). Conflating the two means either episodic events live too long or semantic facts get incorrectly purged.

Consolidation: the bridge between them

The interesting work happens at the bridge. Most semantic facts are not declared by the user; they are inferred from repeated episodic observations. The agent watched you book Felipe's three Saturdays in a row; it consolidated that into the semantic fact "default haircut is Felipe's on Saturday."

This is where dreaming earns its keep. The nightly consolidation pass walks the recent episodic timeline, identifies patterns, and writes new semantic facts. It also demotes semantic facts that have been contradicted (the user has cancelled two haircuts in a row; maybe Felipe's is no longer the default).

The agent that has both layers, plus working consolidation, is the one that feels like it knows you. The agent that has only episodic feels chatty but forgetful. The agent that has only semantic feels rigid and unable to discuss specifics.

A third layer worth naming: core memory

A small set of facts deserve to never be subject to consolidation pressure. Your name. Your immediate family. The two or three things about your work that the agent should always know. These live in core memory: a tiny store, always loaded into the prompt regardless of query, never purged unless explicitly forgotten.

Core memory is what makes the agent feel anchored. The first time you mention your son Sam to an agent and three weeks later it casually references Sam without you re-introducing him, you are seeing core memory at work.

The runtime composition

When the agent receives a new message, it loads memory in this order:

1. Core memory (always loaded, never purged).
2. Recent episodic events (last N from the last K minutes, cross-channel).
3. Semantic facts relevant to the current topic (retrieved via vector search or attribute lookup).
4. Skills relevant to the current task (procedural memory, see the skills post).

The four layers compose into the prompt. The model gets a complete picture: who the user is, what they have been doing recently, what is true about them in general, and how they like things done. The reply is grounded.

Common implementation mistakes

Two we have made and seen others make.

Treating vector search as the whole memory layer. Embedding every memory and using cosine similarity is a useful retrieval primitive. It is not a memory architecture. Vector search retrieves what is similar to the current query; it does not decide what is worth remembering, does not separate timestamped events from steady-state facts, and does not handle the decay/consolidation problem. It is one tool, not the whole system.

Storing semantic facts as long episodic notes. A common shortcut is to write "user prefers morning meetings" as a regular episodic note and rely on retrieval to surface it. This works for a few weeks; eventually the note ages out of the recency window and the agent appears to "forget" the preference. Semantic facts need a separate store with no time-based decay.

What this looks like in practice

When you talk to a serious agent, the four-layer composition is invisible. It just feels like an assistant who knows you. When the layers are missing, the agent's failures show their shape: forgets recent details (no episodic), forgets steady facts (no semantic), forgets you entirely (no core), or does things in the wrong way each time (no skills). Diagnosing which layer is missing is the closest thing to debug intuition for personal-AI products.

Try it

Talk to the agent for a week. Then ask "what do you know about me?" The list will be sorted into facts (semantic + core) and recent context (episodic). That sorted list IS the memory architecture exposed.