Why memory for AI is hard

This page explains why an application does not get memory by calling a language model, what goes wrong when you build memory by saving and searching the chat history, and which of those problems Alvia handles for you.

The examples follow one support assistant at Northwind, a software vendor. Its users talk to it for months, and what they tell it changes.

Inference is not memory

A call to a model API processes the input it is sent and keeps nothing afterward. Anthropic’s Messages API is stateless: “you always send the full conversational history” (Anthropic). OpenAI says the same of its text generation: “each text generation request is independent and stateless” (OpenAI). If Maya, a Northwind customer, told the assistant last week that she wants updates in Slack, today’s call knows it only if your application puts it in the prompt.

What people call memory is several layers, each with its own job. Provider features below are as of October 2026.

Layer	What it holds	Who keeps it
Model inference	Nothing between calls. The model reads its input and returns output.	No one
Conversation history	The messages of one conversation, resent with each call	Your application
Context window	The most input one call can take: instructions, history and retrieved text together	Fixed by the model
Provider conversation state	A conversation’s items, or files the model writes to	The provider, or your storage
RAG and vector retrieval	Passages that are similar to the query	Your pipeline and its index
Persistent application memory	Facts about users, accounts and work that outlive a conversation and serve every later one	A memory system your application writes to and reads
Evolving memory	The same facts, plus how each new one relates to the old: it replaces, adds detail to, repeats, contradicts or relates to one, or is new	A memory system that tracks change

Provider conversation state is real and useful. OpenAI’s Responses API can chain turns with previous_response_id, and its Conversations API keeps a durable conversation that you can reuse across sessions (OpenAI). Anthropic’s memory tool lets Claude create and edit files that your application stores, and memory stores for Managed Agents (in beta) keep versioned text files for an agent. These keep transcripts or files. Deciding that one fact replaced another is left to your code, to the model’s edits or to a later consolidation job. See Alvia compared with other approaches.

The ChatGPT and Claude apps appear to remember because the product around the model does that work: it saves what you said, chooses what matters and puts it back into later prompts. An application that calls the API inherits none of it.

Saving the chat history is not a memory system

The first design most teams build looks simple:

1. Save every message in a table.
2. Embed each message in a vector index.
3. On each turn, retrieve the ten most similar messages and add them to the prompt.

It looks fine in a demo. Over months of Northwind conversations, it fails in production in these ways:

• Duplicates. Maya says “email is best for me” in six tickets. Six near-identical messages fill the retrieved slots, and the one fact that differs is pushed out.
• A changed fact returns both versions. In June Maya says “use our Slack channel from now on; email gets lost.” Both the March and June messages match “How does Maya want updates?” The model gets both and must guess which holds. Sorting by newest helps only until the next problem.
• A contradiction. One agent’s note says Maya wants a phone call for outages; another says she asked never to be called. Neither replaces the other. Someone has to see that they disagree.
• When a fact was stated. In October you import a June call in which Maya said “starting next week, use Slack.” Stored with today’s date, the fact looks newer than a September message that changed it again, and “next week” points to the wrong week. A memory needs the time a fact was stated, not the time it reached you.
• Scope. Maya prefers Slack; her colleague Sam still wants email; another customer must never appear in Northwind’s answers. A similarity search over one index does not know whose fact is whose unless every write and every read carries the same scope.
• The context budget. After 300 tickets, the relevant messages no longer fit. Retrieve fewer, and the answer is missing; retrieve more, and the prompt is long, slow and costly.
• Deletion. Maya leaves Northwind and asks to be forgotten. Her words live in the message table, the vector index, summaries and cached prompts. Deleting one row reaches none of the copies.
• Everything is kept. The table also holds the card number she pasted and the health reason she gave for missing a call. Nothing decided what was worth remembering.
• Cost and latency. The usual fixes run a model over everything: summarize the history on every turn, or ask a model to reconcile twenty retrieved messages before each answer. The cost grows with the history, and the user waits for it.

The prototype and the system

A dependable memory system is a different engineering problem from the prototype:

The prototype	What production needs
Save every message	Decide what is a fact, extract it and merge repeats
Retrieve the most similar messages	Rank what applies now above what was replaced, and fit the result to a budget
A created_at column	The time each fact was stated, relative dates and backfills that arrive out of order
A user_id column	One scope on every write and read, and new facts compared only within it
Delete a row	Remove the fact from every index, summary and cache it reached
Check a few answers by eye	Test that a changed fact returns the new one, and test again when you change models
One process	Background queues, retries, backfill throttling and the cost of each model call

What Alvia handles and what stays with you

Alvia resolves how each new fact relates to what it holds when the fact arrives, not on every read. Each problem above maps to a part of the service:

Problem	What Alvia does	Read more
Duplicates	Formation takes out each durable fact. A fact Alvia classifies as a duplicate stores nothing new.	Memory formation
A changed fact	A fact classified as a replacement becomes the memory’s next version, and reads return it. The old text stays in the memory’s history.	Memory formation
A contradiction	Alvia keeps both and links them as contradicting, so context reports the disagreement instead of guessing.	Memory formation
When a fact was stated	Each memory carries occurred_at, the latest time among the statements that formed or revised it. Relative dates such as “yesterday” resolve on that day, and Alvia writes the dates they name into the text.	Point-in-Time Reasoning and change over time
Scope	Writes store memories under the exact scope you send, and formation compares a new fact only with memories in that scope. Reads filter by the scope fields you send.	Projects and scope
The context budget	Context returns one to three sentences; recall returns the single best memory; search returns a ranked list you size with limit.	Read memories
Ranking over time	Each category’s ranking boost fades with age, quickly for events and slowly for preferences. Feedback and favorites adjust rank.	Point-in-Time Reasoning and change over time
Deletion	Forget removes a memory from every read for good. Deleting the project erases all the text sent to it.	Forget a memory
Everything is kept	A project’s memory policy excludes information by keyword and by meaning before it becomes memory.	Memory policies
Cost and latency	Relationships are formed once, in the background, after a write returns 202. A repeated question returns quickly.	Production checklist

Every memory keeps its versions and how each was classified. Your application reads a memory’s history, versions and relationships through the API, and the dashboard shows each version, how Alvia classified it and why. Every new account has a Demo Project whose sample memories include replaced facts and a contradiction.

Your application keeps these:

• Identity. Your sign-in decides who the user is.
• Authorization. Your server decides which project and scope a request may use. The project is the access boundary: every key reads every memory in its project, and scope keeps each user’s memories together. See Security.
• Authoritative records. Plans, balances, seat counts and anything else your database owns stay there. Read them from the database, not from memory.
• What to send. You choose what Alvia sees. Remove data that must never leave your systems before you send it.
• Acting on the answer. Alvia returns context; your application or agent decides what to do.

Related pages

• Alvia compared with other approaches
• Memory formation
• Point-in-Time Reasoning and change over time
• Projects and scope
• Add memory to an LLM app