PAPER DEEP DIVE
Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought
LLM memory is widely used in long-term conversation understanding. Agentic memory systems often use mutation, structural organization, learned retrieval strategies, or other complex operations. Is this complexity always necessary? We study Raw-RAG, a naive RAG baseline using dense retrieval and reranking over raw history to supply evidence to a reader LLM. Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, Raw-RAG ranks first among the compared memory systems in six of seven benchmark-model settings, outperforming the strongest compared agentic memory system by 0.13-5.53 points. We investigate why raw-history retrieval remains competitive. Evidence analysis shows that its higher coverage of necessary evidence can offset lower utilization. Useful evidence includes both answer values and the relations and dependency paths needed to derive them. Even complete access to transformed memory stores leaves an accuracy gap relative to raw evidence, highlighting the importance of preserving raw information. These findings challenge the necessity of complex agentic memory for the evaluated conversation tasks and establish naive RAG as an essential baseline.
One-sentence summary
The paper treats raw-history retrieval with dense recall, cross-encoder reranking, and a capable reader as a diagnostic baseline. Raw-RAG wins six of seven benchmark-model settings, and the evidence suggests that complex memory systems often lose information during ingestion rather than solving a problem that raw evidence cannot support.
Figure 1 — Accuracy across seven benchmark-model settings and cumulative sequential ingestion on BEAM-1M. Raw-RAG remains competitive while avoiding LLM-based memory extraction, summarization, graph construction, and mutation.
1. Reframing the question around evidence access
Long-running assistants cannot place every prior conversation inside the active context window. They need a policy for deciding what survives, how it is represented, and how relevant history is returned. Contemporary agentic memory systems answer this problem with extraction, consolidation, graph construction, deletion, learned retrieval, or recurrent organization. These mechanisms are often treated as necessary for temporal updates, preferences, and multi-session reasoning.
This paper asks whether that complexity is necessary under the evaluated conditions. Its alternative is deliberately narrow. Raw-RAG serializes conversation turns with timestamps, speakers, text, and available image captions. It splits the source into overlapping windows, indexes the windows, retrieves candidates with dense similarity, reranks them with a cross-encoder, and gives the selected raw evidence to a reader model. No ingestion-time LLM decides which facts should survive.
The design separates two explanations for the success of agentic memory. One possibility is that readers genuinely benefit from an explicit, maintained state. Another is that prior RAG baselines were too weak: they returned small numbers of coarse passages, omitted reranking, or failed to recover complete turns. If the second explanation is correct, a strong raw-retrieval baseline may close much of the reported gap without reproducing the full memory-management pipeline.
The paper begins with a direct audit of Mem0. On LoCoMo, UPDATE and DELETE account for only 4.39% of 33,902 emitted write decisions. On LongMemEval, they account for 2.46% of 428,858 decisions. Persisted text-changing mutations are rarer still: 1.90% on LoCoMo and 1.34% on LongMemEval. Many proposed updates repeat existing content, fail schema checks, or leave the stored text unchanged.
An intervention replaces UPDATE with append-only ADD and DELETE with NONE. LoCoMo accuracy rises from 67.36% to 69.42%, with no category-level decline. The result does not establish that deletion is useless in every application. It shows that, in this benchmark and configuration, explicit replacement did not help the reader. Retaining old evidence and allowing the reader to resolve chronology avoided some of the information loss caused by aggressive mutation.
A second diagnostic supplies MEME readers with phase-visible raw evidence in chronological order. Retrieval errors are removed, but deletion, cascades, and invalidation still require state reconstruction. DeepSeek-V4-Flash reaches 82.32% on cascades and 91.54% on invalidation. This oracle-style result supports a limited claim: a sufficiently capable reader can derive difficult transitions when the source evidence is present.
2. The Raw-RAG pipeline
Let the available history be $H=(m_1,\ldots,m_T)$. Each message retains its date, speaker, and content; a missing date is marked as unknown rather than invented. The serialized history is divided into windows of 256 tokens with 32 tokens of overlap. The offset between adjacent windows is therefore:
$$W_{\mathrm{stride}}=W_{\mathrm{window}}-W_{\mathrm{overlap}}=256-32=224.$$
Every window remains linked to the original messages it covers. If a selected window cuts through a turn, the system can restore the complete source turn before showing evidence to the reader. Overlap protects against the case where an important event lies exactly at a boundary; turn expansion prevents the reader from seeing a question or answer without its surrounding conversational context.
The indexing representation does not replace the source text with facts or summaries. The question is embedded, and cosine similarity selects 128 candidate windows from the conversation-scoped index:
$$D_q=\operatorname{Top}_{128}\{\cos(e(q),e(c)):c\in\operatorname{chunks}(H)\}.$$
Here, $e(\cdot)$ is the embedding encoder and $\operatorname{chunks}(H)$ is the set of raw windows. The large candidate set is a recall stage, not the final reader context. It gives the reranker room to recover passages that are relevant to the full question even when their first-stage similarity is imperfect.
A cross-encoder then reads each question-window pair jointly and produces a relevance score. Only the highest-scoring 15 windows enter the reader input:
$$S_q=\operatorname{Top}_{15}\{r(q,c):c\in D_q\}.$$
The function $r(q,c)$ models interactions between question tokens and passage tokens. This is materially different from comparing two independently produced embeddings: a cross-encoder can match a date, entity, relation, or condition in the question against the corresponding passage text. The ablation confirms that this stage matters, removing it reduces MemoryAgentBench Total by 3.92 points.
Selection and presentation are separate operations. Raw-RAG first chooses windows by relevance, then expands them into complete turns with timestamps, speakers, and separators. MEME restores source order after selection so the reader can inspect temporal progression. Ordinary MemoryAgentBench conversations disable display-time turn expansion, while two fact-consolidation sources index each fact as a complete message. The system is “raw” in the sense that it does not irreversibly extract or rewrite the evidence, not in the sense that formatting is absent.
flowchart LR
H["raw conversation history
timestamps and speakers"] --> S["serialize source-preserving history"]
S --> C["256-token windows
32-token overlap"]
C --> I["flat dense index"]
Q["question"] --> E["query embedding"]
E --> D["dense Top-128
cosine candidates"]
I --> D
D --> R["cross-encoder rerank
Top-15 windows"]
R --> T["expand selected windows
to complete source turns"]
T --> P["assemble evidence in
relevance or chronological order"]
P --> A["reader resolves chronology
deletion and dependency state"]
A --> O["answer"]
style S fill:#e0f2fe,stroke:#0284c7
style D fill:#fef3c7,stroke:#d97706
style R fill:#dcfce7,stroke:#16a34a
style A fill:#fce7f3,stroke:#db2777
Figure 3 — Build, store, and recall paths for EverMemOS, Memora, and Raw-RAG. The agentic systems create organized memory representations, while Raw-RAG keeps source windows and relies on dense recall, reranking, and turn recovery.
3. Coverage explains more than conditional answering skill
The paper separates whether the context contains sufficient evidence from whether the reader answers correctly once sufficient evidence is available. For question $i$ and system $s$, let $C_{is}\in\{0,1\}$ denote evidence sufficiency and let $Y_{is}\in[0,1]$ denote the answer score. Coverage and utilization are:
$$\widehat C_s=\frac{1}{N}\sum_{i=1}^{N}C_{is},\qquad \widehat U_s=\frac{\sum_{i=1}^{N}C_{is}Y_{is}}{\sum_{i=1}^{N}C_{is}}.$$
Coverage is the fraction of questions for which the retrieved context can support the answer. Utilization is accuracy conditioned on that sufficient set. The distinction prevents a system from looking unusually good merely because its sufficient-evidence subset consists of easy cases. It also prevents a system with broad recall from being described as universally better at reasoning when its reader still fails on some of the evidence it receives.
On MemoryAgentBench, coverage spans 19.07% to 45.29% across seven systems, while utilization spans only 80.00% to 89.71%. Raw-RAG reaches the highest coverage, 45.29%, but its 84.43% utilization is not the highest. Its aggregate advantage therefore comes mainly from placing adequate evidence into the context more often, not from dominating every conditional reasoning step.
MEME repeats the pattern. Raw-RAG covers 76.35% of questions with 73.76% utilization. MemMachine covers 70.37% with 82.66% utilization. The systems arrive at similar overall performance through different operating points: Raw-RAG exposes more evidence and accepts weaker conditional use, while MemMachine exposes less evidence but uses the selected evidence more effectively.
| System | MemoryAgentBench coverage | MemoryAgentBench utilization | MEME coverage | MEME utilization |
|---|---|---|---|---|
| MemU | 19.07% | 89.37% | 35.69% | 84.20% |
| MemMachine | 21.54% | 84.30% | 70.37% | 82.66% |
| EverMemOS | 25.11% | 80.00% | 47.64% | 76.86% |
| MemOS | 34.48% | 85.85% | 62.12% | 66.67% |
| EMem | 34.72% | 89.71% | 50.08% | 86.55% |
| Mem0 | 40.03% | 82.39% | 63.22% | 65.51% |
| Raw-RAG | 45.29% | 84.43% | 76.35% | 73.76% |
Useful evidence also includes relations, not only answer values. For tracking and cascade tasks, a context may mention the right values but omit their chronology, dependency rule, or upstream trigger. The paper measures path coverage among questions whose answer values were already covered:
$$P(\mathrm{path}\mid\mathrm{result}) =\frac{N(\mathrm{path}\land\mathrm{result})}{N(\mathrm{result})}.$$
Raw-RAG achieves 62.34%, followed by MemMachine at 56.70%; MemU reaches 9.62%. Two systems may therefore retrieve the same isolated value while differing sharply in whether the reader can derive the current state from it. This distinction is particularly relevant to agent harnesses, where an action may depend on an event sequence, a conditional relation, or a trigger that changed an earlier fact.
4. Complete-store access does not erase the value of raw evidence
Broader retrieval can fail because evidence is absent from storage or because it is present but not retrieved. The paper tests the first possibility with a gold-informed complete-store diagnostic on MEME. It compares native retrieval, privileged access to every exported persisted memory object under a nominal 5,000-token budget, and a raw-evidence control that selects annotated source sessions with the same gold information. Both privileged conditions receive a value-free task policy.
For the complete-store selector, normalized gold phrases and question words produce a ranking over a stored object $m$:
$$s(m)=\sum_{h\in H(m)}\left(4+\min(4,|h|/3)\right) +2\frac{|\operatorname{words}(m)\cap Q|}{\max(1,|Q|)}.$$
The function counts phrase hits and a small question-word overlap term; ordinary retrieval similarity is deliberately absent. This gives transformed stores a favorable test. Even under that access, Raw-RAG and MemMachine recover to 57.54% and 57.46%, while raw annotated evidence reaches 61.60%. Removing the packing ceiling leaves all seven stores 7.58 to 29.38 points below the raw control.
The authors do not claim that this residual gap isolates ingestion loss. Ordering, packing, task-policy framing, and lost-in-the-middle effects remain confounders. The defensible conclusion is narrower: source-preserving stores recover closest to the raw-evidence control, while complete access to transformed stores cannot generally reconstruct information that was removed during ingestion.
5. Main benchmark results
The evaluation spans four benchmarks. LoCoMo contributes 1,540 non-adversarial questions from ten conversations. LongMemEval contributes 500 examples covering recall, temporal reasoning, preferences, and updates. MEME contributes 1,188 phase-specific questions from 100 episodes, including exact recall, aggregation, tracking, deletion, cascade, and invalidation. MemoryAgentBench contributes 2,071 questions across 14 sources and four capabilities: accurate retrieval, test-time learning, long-range understanding, and conflict resolution.
Comparators include EverMemOS, MemOS, EMem, SimpleMem, MemU, MemMachine, Mem0, Memora, Nemori, MemoryOS, A-MEM, RecMem, and Full-context. Where possible, Raw-RAG uses the same reader, embedding model, evaluation history, and scoring procedure as the corresponding prior comparison. Main rankings exclude Full-context because it performs no retrieval and cannot consume every long history within a fixed context window.
| Benchmark | Reader | Raw-RAG | Best compared memory system | Difference |
|---|---|---|---|---|
| LoCoMo | GPT-4o-mini | 89.07 | MemMachine 88.12 | +0.95 |
| LoCoMo | GPT-4.1-mini | 93.70 | EverMemOS 93.05 | +0.65 |
| LongMemEval | GPT-4.1-mini | 87.53 | Memora 87.40 | +0.13 |
| MEME | GPT-4o-mini | 39.30 | MemMachine 37.06 | +2.24 |
| MEME | DeepSeek-V4-Flash | 43.19 | MemMachine 44.71 | -1.52 |
| MemoryAgentBench | GPT-4o-mini | 48.32 | Memora 43.84 | +4.48 |
| MemoryAgentBench | DeepSeek-V4-Flash | 53.41 | Memora 47.88 | +5.53 |
Raw-RAG ranks first in six of seven settings and beats the strongest compared memory system by 0.13 to 5.53 points. The exception is MEME with DeepSeek-V4-Flash, where MemMachine scores 44.71 and Raw-RAG scores 43.19. The exception is analytically important: reader choice can change the optimum between raw evidence and managed state, so “naive RAG always wins” would overstate the finding.
For the 470 answerable LongMemEval questions, the three repeated judge decisions can be written as:
$$\mathrm{Acc}_{\mathrm{ans}} =\frac{100}{3|Q_{\mathrm{ans}}|} \sum_{i\in Q_{\mathrm{ans}}}\sum_{r=1}^{3}y_{ir}, \qquad |Q_{\mathrm{ans}}|=470.$$
MemoryAgentBench first combines ReDial with the average of five in-context-learning sources, then averages the four capability scores:
$$\mathrm{TTL}=\frac12\left(\mathrm{ReDial} +\frac15\sum_{j=1}^{5}\mathrm{ICL}_j\right), \qquad \mathrm{Overall}=\frac{\mathrm{AR}+\mathrm{TTL}+\mathrm{LRU}+\mathrm{CR}}{4}.$$
The results are not uniformly high across every subskill. MEME cascade scores remain low in absolute terms for both readers. Raw-RAG’s advantage is therefore an aggregate result over a broad task suite, not evidence that a flat raw index solves every dependency graph. It means that, on these benchmarks, a strong retrieval baseline supplies enough evidence to outperform the compared managed systems more often than not.
6. Ablations identify where the advantage comes from
The default MemoryAgentBench run uses 256-token windows, 32-token overlap, 128 dense candidates, and 15 reranked windows. Removing the cross-encoder lowers Total from 48.32 to 44.40. Returning only one item lowers it by 11.73 points. Final depths between 5 and 60 remain within 1.3 points of the default, while 120 items lose 14.57 points, mainly in long-range understanding and conflict resolution.
Changing chunk size from 128 to 1,024 tokens does not significantly change the aggregate. This supports a retrieval design in which moderate depth and reranking matter more than a single magic window length. It also shows why a weak RAG baseline is a poor counterfactual: a one-item, no-rerank pipeline creates a gap that can be misread as evidence for an entirely different memory architecture.
Three memory-management additions are tested on the same benchmark. Atomic-fact extraction drops Total to 28.70, raw-summary augmentation reaches 42.51, and graph expansion over raw windows reaches 42.54. Atomic extraction is especially harmful to test-time learning, falling from 47.68 to 15.37. The paper suggests that classification demonstrations require recognizable input-label pairings and boundaries; replacing them with isolated facts damages evidence structure even when some factual content remains.
| MemoryAgentBench setting | AR | TTL | LRU | CR | Total | Delta |
|---|---|---|---|---|---|---|
| Raw-RAG default | 67.85 | 47.68 | 31.74 | 46.00 | 48.32 | — |
| Without reranker | 59.13 | 44.29 | 29.66 | 44.50 | 44.40 | -3.92 |
| 128-token / 16-overlap chunks | 70.50 | 46.89 | 28.93 | 46.00 | 48.08 | -0.24 |
| 512-token / 64-overlap chunks | 65.68 | 49.27 | 32.49 | 46.00 | 48.36 | +0.04 |
| Atomic facts | 57.25 | 15.37 | 21.17 | 21.00 | 28.70 | -19.62 |
| Raw summary augmentation | 67.53 | 45.89 | 29.62 | 27.00 | 42.51 | -5.81 |
| Graph-expanded raw windows | 65.42 | 46.49 | 31.74 | 26.50 | 42.54 | -5.78 |
These results do not prove that every structured or summarized memory is ineffective. They prove that three concrete additions fail to improve one well-specified benchmark and reader. A different task, a weaker model, a tool-facing state query, or a larger context budget could reverse the trade-off. The methodological requirement is to compare against a strong raw baseline and identify the mechanism by which the added representation helps.
7. Efficiency and the location of cost
Raw-RAG shifts work away from ingestion. On the common 10-episode, 110-question MEME timing slice, Add-to-ready takes 50.32 seconds and Search takes 255.59 seconds, for 305.91 seconds total. Answer generation and evaluation are excluded. The next-fastest system takes 1,561.97 seconds, so Raw-RAG is about 5.1 times faster; EverMemOS takes 9,301.48 seconds, making Raw-RAG about 30.4 times faster on this slice.
The BEAM-1M experiment streams a 1,710-message conversation approaching one million tokens into each system with one external worker. Cumulative Add time at the endpoint is about 0.33 hours for Raw-RAG, 3.25 hours for MemOS, and 26.95 hours for Memora. The advantage is primarily construction latency, not total question-answering cost.
Query-time reasoning can be expensive because the reader must inspect raw evidence and resolve state on every request. Raw storage also grows linearly with history and increases privacy exposure. A deployment with frequent queries, strict latency limits, or audited current-state reads may still benefit from maintaining structured state. The paper’s result changes the burden of proof: complexity must show benefits beyond a strong raw baseline, including storage, deletion, security, and query-cost trade-offs.
8. Implementation correspondence
The paper does not provide a code link in its main text. A public repository matching the title, abstract, and evaluation design is available at Litmeb/RAW-RAG under the MIT license. It should be described as a public implementation corresponding to the paper’s method and experiments, not as an explicitly designated unique official release. The inspected commit is 95639191e9547219b426710e50fe1a36235e26fc.
The repository’s methods/EverCore/evaluation/config/systems/rag.yaml configures 256-token windows, 32-token overlap, dense top-128 candidates, and top-15 reranked evidence. rag_adapter.py implements token-window creation in _create_token_chunks, local cosine retrieval in _get_local_embedding_candidates, and cross-encoder HTTP reranking in _rerank_candidates.
The reader path is also explicit. config/prompts.yaml defines answer_prompt_cot, which passes raw evidence and the question to the answering model and asks it to extract relevant memories, connect entities across memories, calculate temporal references, and check contradictions. The implementation therefore clarifies that “raw” refers to preserving source evidence, not to removing all reasoning scaffolds from the reader.
The repository includes scripts for LoCoMo, LongMemEval, MEME, MemoryAgentBench, PersonaMem-v2, the three memory-management ablations, and BEAM ingestion timing. It also documents an independent GoodMem reproduction: the reported LoCoMo scores are 88.68 for GPT-4o-mini and 93.48 for GPT-4.1-mini, close to the paper’s 89.07 and 93.70. This external run reduces, but does not eliminate, concern that the result depends on one implementation.
Figure 2 — Mem0 write actions and persistent mutation rates on the LoCoMo and LongMemEval conversations. Dashed lines are decision-weighted means; effective text-changing updates and deletions are rare relative to all emitted decisions.
9. Limitations
The authors explicitly limit the evidence to offline conversation on public benchmarks. The results do not establish behavior for interactive, multimodal, or broader operational settings. Real agent systems include streaming sensor data, asynchronous tool results, multiple users, permission boundaries, and physical actions whose cost of error differs from a benchmark answer.
Raw-history storage raises privacy and retention problems. The paper states that future evaluations must account for explicit privacy and deletion constraints. A system that preserves every source turn for recoverability may conflict with data minimization and verified deletion. Merely instructing a reader not to use an old value is not equivalent to removing it from storage.
State reconstruction depends on the reader. The phase-visible MEME oracle varies substantially across models, and the paper leaves open whether explicit state maintenance helps weaker readers or reduces repeated reasoning. Raw evidence can be advantageous for capable readers while overwhelming a model that struggles to order events, reconcile conflicts, or apply dependency rules.
Raw storage grows with history. Fixed candidate and reader budgets can miss dispersed, low-similarity, or multi-hop evidence. Turn expansion and overlapping windows can repeat text and consume context. The paper’s 5,000-token sensitivity checks show that Raw-RAG remains competitive with shorter evidence on several benchmarks, but longer interactive workloads and bounded storage may favor different trade-offs.
Coverage and utilization are estimated on system-specific sufficient-evidence sets. The authors note that this prevents a fully controlled comparison of readers across systems. Utilization conditions on a different denominator for each system, so it should be used to diagnose where a system loses performance rather than as an isolated causal ranking.
The complete-store analysis is also an operational diagnostic rather than an information-theoretic decomposition. Gold-informed selection, packing order, task policy, and reader position bias affect results. The authors therefore avoid attributing every point of the raw-store gap to ingestion-time information loss.
Finally, the tasks ask for answers, not physical action. A robot-facing harness needs to connect memory with sensor provenance, action outcomes, safety constraints, permission checks, and real-time state. Better conversational recall is relevant infrastructure, but it is not sufficient evidence that a memory policy is safe or executable in a physical environment.
10. Conclusion
The paper’s central contribution is not a claim that every agent should use a flat vector index. It is a stronger counterfactual for evaluating memory. Raw-RAG demonstrates that preserving source text, retrieving broadly, reranking carefully, and exposing complete turns can outperform many sophisticated systems on four long-term conversation benchmarks. Coverage and path analyses explain why: complex ingestion can remove not only answer values but also the relationships needed to use those values.
For an agent harness, the practical evaluation checklist follows directly. A memory system should report gains over a well-specified raw baseline, separate evidence coverage from reader utilization, test dependency and temporal paths, measure both ingestion and query costs, and define deletion and privacy semantics. Complexity is justified only when those measurements identify a concrete benefit that raw retrieval cannot provide.
Key takeaway
A complex memory system must show more than a polished state representation. It must preserve enough evidence, relations, and temporal context to beat a strong raw-history retrieval baseline at a defensible cost.



