The biggest hole in the open source AI stack is memory, and it matters more than the model

Short answer: In 2026 you can swap the model behind your product with a one line change. You cannot swap in memory, because the open stack does not have a good one. Self built memory on a vector database fails in six predictable ways: it stores text instead of knowledge, grows without bound, goes stale, has no access control, cannot show its sources and cannot prove deletion. The field's own reports call several of these unsolved. That gap is where a startup's real advantage lives.

The model got solved. Memory did not.

Something odd happened in the open source AI stack this year. The hardest part, the model, became the easy part.

DeepSeek, Qwen, GLM, Kimi, Mistral and gpt-oss put frontier class intelligence behind an open licence at a fraction of the price. Routers like OpenRouter made switching between them trivial. Inference providers like Cerebras made them fast. If your product needs a model, you have a dozen good choices and you can change your mind on a Tuesday.

Then you try to make the model know something about your company, and the stack goes quiet.

The standard advice is still what it was in 2023. Chunk your documents, embed them, put them in a vector database, retrieve the top few chunks and paste them into the prompt. Every open source agent framework has a version of this. Every startup builds one. And nearly every one of them hits the same wall within six months.

Six ways self built memory fails

1. It stores text, not knowledge. A chunk of a document is not a fact. "The lender's rate is 6.2%" and "the client approved the draft on 3 March" and "this is how we run a completion" are different kinds of knowledge that need to be stored, updated and retrieved differently. A bag of similar looking text chunks treats them all the same, so the answer to "what did we agree" comes back as a paragraph that mentions agreeing.

2. It grows without bound and retrieval gets worse. More memory sounds better. In practice, an agent that never forgets accumulates contradictions, and a one off detail from last month outranks a durable rule from last year because it is more recent or more similar. One engineering team building an open source memory server described the moment recall got worse after adding memory: five overlapping notes about the same bug, each slightly wrong, splitting the signal. One production benchmarker put it more bluntly: none of the systems solve the question of what to remember and what to forget. That is not a fringe view. Mem0's own State of AI Agent Memory 2026 report lists staleness among the field's open problems.

3. It goes stale and gets confidently wrong. A memory about a customer's role is accurate until they change jobs. Then it is retrieved with high confidence and it is wrong. Decay handles trivia. Staleness in the memories that matter most is, by the industry's own account, unsolved at the tooling level.

4. It has no idea who is allowed to see what. Vector databases retrieve by similarity. They do not know that this chunk came from a matter the person asking is walled from, or a fund vehicle they do not work on. Access control has to be bolted on afterwards and it usually is not. One analysis of eight leading agent memory frameworks found that all of them lacked enterprise governance: no lineage, no entity resolution, no glossary.

5. It cannot show where an answer came from. When the model answers from retrieved chunks, the chain from source document to answer is usually lost. In a regulated business an answer without a source is a liability. In any business it is an answer you cannot check.

6. It cannot prove deletion. A customer invokes their data rights. Their information has been chunked, embedded and scattered across an index that was never designed to give it back. "We believe it is gone" is the most that many teams can say. It is not what a regulator wants to hear.

Why this is where the advantage is

Here is the strategic point. If every startup can rent the same models at the same price, the model is not a differentiator. What separates a product that customers keep from one they churn from is how well it knows them, their documents, their history and their decisions. That is memory. And because the open stack does not solve it, most competitors either do not have it or have a fragile home built version that fails in the six ways above.

The startup that gets memory right has an advantage that compounds. Every document, conversation and decision after setup makes its product better. The model can be swapped underneath and nothing is lost.

The open memory tools, fairly assessed

There are serious open source efforts. Letta, from the MemGPT lineage, treats memory like an operating system and leads the open field on the LongMemEval benchmark at around 83%. Mem0 is the most widely adopted developer memory library, and its managed version reports strong benchmark results. Zep, Cognee and others take graph based approaches.

They are good tools for the problem they set out to solve: memory for a developer's application. Most were not built for a company. They assume one developer, one app, code to integrate them, and they leave governance, sourcing, sharing across a team and provable deletion to you. For a consumer app that is fine. For a business, and certainly for a regulated one, it is the whole problem.

What a governed memory layer does differently

A governed memory layer starts from the six failures and designs against each.

Typed records instead of chunks: facts, events and procedures stored and retrieved as what they are.

Bounded, structured memory that consolidates rather than accumulates, with staleness handled by design rather than hoped away.

A source on every record, so every answer carries the document, email or commit it came from.

Access enforced on every record, by role, so the answer a person gets is drawn only from what they are allowed to see.

An audit log of every read and write.

Deletion at the record level that can be demonstrated.

And it sits one layer up from any model. Ask it from a cheap open model on Cerebras or from a frontier model; the memory is the same, and it does not care which model is asking.

How OctaMem does this

OctaMem is a governed memory layer built for organisations, and it is model agnostic by design. It connects to Drive, Outlook, Teams, GitHub and SharePoint through a desktop app, turns everything into typed, source linked memory, scopes access by role down to the record, logs every read, and can prove deletion. Any model, router or agent connects through MCP or the API.

On the LoCoMo long term memory benchmark OctaMem scores 93.51%, 1,440 of 1,540 questions, at about $0.00076 per question, and we publish the questions we get wrong. Benchmark numbers across vendors are only comparable when the methodology is the same, so read the methodology, ours included, before you compare. Team plans start at $199 a month.

Frequently asked questions