Memory Tax: Dissecting the Net Value of Agent Memory

One-Sentence Summary

Using naive retrieval-augmented generation over raw interaction history as the reference, the paper separates the net value of agent memory into a measurable memory benefit and memory tax, then shows across three main benchmarks, experiential memory, and world-model memory that dedicated memory can recover queries missed by RAG while also losing queries that RAG already answered.

Why This Problem Matters

Long-term memory has become a standard part of long-horizon agent infrastructure. Systems summarize conversations, extract facts, maintain hierarchical stores, build graphs, or train policies so that useful information survives beyond a finite context window. This has produced a natural architectural race toward more elaborate memory, with the implicit assumption that richer organization should improve the answers delivered to users.

The missing control is deceptively simple: what if the agent retrieves directly from its raw history? Raw-history RAG performs little or no memory construction. It chunks past observations and dialogue, retrieves relevant chunks for the current query, and leaves the evidence largely intact. A memory system may organize that history much better, but it also changes it. The paper argues that prior comparisons often measure one memory design against another while leaving the stronger, simpler retrieval reference underexamined.

That omission entangles several capabilities. A memory result may improve because its architecture exposes better evidence, but it may also improve because the experiment gives it a larger retrieval budget, a stronger embedding model, a different answer prompt, or a more favorable judge. Once those factors are controlled, the remaining comparison concerns the marginal value of memory construction itself.

The authors do not argue that memory is useless. Their argument is that memory is a double-edged transformation. Extraction and organization can expose implicit preferences, connect distant events, or compress a workflow into a reusable abstraction. The same operations can erase a qualifier, split evidence across units, replace a precise observation with a lossy paraphrase, or make the final system dependent on the model that performed ingestion. Net value is the balance between these intended and unintended consequences.

Formal View of the Memory Pipeline

The interaction history is represented as an ordered set $\mathcal{H}=\{h_{1},\ldots,h_{T}\}$, where each item may contain a user message, agent action, environment observation, or another record. A memory system applies an ingestion function to produce a persistent representation $\mathcal{M}=I(\mathcal{H})$. The function $I$ is where summaries, extracted facts, graph edges, life-cycle rules, or learned write policies enter the pipeline.

At query time, the system retrieves context $\mathcal{C}=R(q,\mathcal{M})$, after which the generator produces $\hat{y}=G(q,\mathcal{C})$. A RAG reference follows the same retrieval and generation interface but minimizes the transformation imposed on history. This shared notation makes the controlled comparison explicit: the benchmark can hold $q$, $G$, judging, and broad retrieval budgets fixed while varying $I$ and the access mechanism inside $R$.

Memory benefit is defined per query as the case where the memory system is correct and RAG is incorrect. Memory tax is the opposite case: RAG is correct while the memory system is incorrect. Both quantities are relative to a named RAG configuration, backbone, benchmark, and judge. The paper does not replace these empirical set differences with a fabricated arithmetic equation, so the appropriate reading is that benefit and tax are observed correctness flows over a query population.

Net value is the balance of those flows. A average score can hide the underlying movement. Two systems may finish at the same accuracy while one expands the answerable set and the other merely preserves RAG successes. The Venn comparisons and per-query replays are therefore central evidence, not decorative diagnostics. They reveal where a system gains capability and where it damages an existing one.

Experimental Design

The main study spans eleven memory systems in four families. A-Mem, SimpleMem, and MemOS form the flat-store group. MemoryOS, LightMem, and OpenViking represent hierarchical or life-cycle designs. HippoRAG2, PlugMem, and MAGMA are graph-based. Mem-T and Memory-R2 use trained memory policies. The grouping is broad enough to test whether the tax is specific to one implementation style.

LongMemEval-s contributes 500 questions with corpora averaging about 115,000 tokens and mostly short answers. PersonaMem-v2 contributes 40 personas and 934 four-way multiple-choice questions at roughly 123,000 tokens per corpus. MINT-Eval contributes 60 corpora and 1,557 questions at about 139,000 tokens per corpus, covering multi-turn dialogue, state tracking, GitHub commits, and wiki revisions. The three tasks range from direct recall to personalization, temporal updates, and ordered state reasoning.

Qwen3.5-4B, Qwen3.6-27B, and GPT-5.6 Luna low serve as answer backbones. Every system shares the same embedding model, answer prompts, and LLM judge, so score differences are intended to come from memory construction and retrieval rather than answer conventions. The authors align retrieval scope by measuring the fraction of each system's store exposed to the model through retrieval and adjusting top-k so that memory systems are not accidentally starved of evidence relative to one-round RAG.

This budget control addresses a real confound. Five retrieved summaries and five retrieved atomic facts can differ by an order of magnitude in token volume. Holding top-k fixed would therefore hold the nominal number of units fixed while leaving the actual evidence budget uncontrolled. The appendix also documents system-specific adjustments, including a smaller SimpleMem window, the use of both user and assistant turns in LightMem, and changes to MemOS post-processing. Those choices are followed by checks intended to show that they favor rather than penalize the memory baselines.

flowchart LR
    H["Raw history H"] --> R["Naive RAG reference"]
    H --> I["Memory construction I of H"]
    Q["Query q"] --> RR["Retrieve raw chunks"]
    Q --> MR["Retrieve memory units"]
    RR --> A1["Answer with RAG"]
    MR --> A2["Answer with memory"]
    A1 --> V["Per-query correctness comparison"]
    A2 --> V
    V --> B["Benefit: memory succeeds where RAG fails"]
    V --> T["Tax: RAG succeeds where memory fails"]
    B --> N["Net position on the evaluated corpus"]
    T --> N

Main Results: Added Complexity Does Not Guarantee Added Accuracy

The headline finding is not simply that RAG wins. It is that many systems lose substantial accuracy after investing more construction and retrieval work. With Qwen3.6-27B on LongMemEval-s, two-round RAG reaches 81.4, while SimpleMem reaches 73.6, MemoryOS reaches 73.4, Mem-T reaches 75.0, and OpenViking falls to 58.8. On MINT-Eval, RAG reaches 33.1; MemOS reaches 19.7, SimpleMem 19.9, and MAGMA 27.0. PlugMem is the narrow exception in that slice, ahead by 0.1 point at 33.3.

PersonaMem-v2 is less one-sided. MemOS and Memory-R2 beat RAG by 1.0 and 2.0 points on Qwen3.5-4B, which is credible evidence that personalization memory can expose useful preference evidence that raw chunks leave implicit. The same systems are weaker in other combinations, however. The authors use that variation to investigate preservation, access, and backbone sensitivity instead of treating every negative result as a flawed implementation.

Figure 1 places accuracy, token cost, and memory-store size on one plane. A large bubble means that a system maintains a large memory, but it does not mean that the extra storage yields a stronger answer. Memory systems often occupy a more expensive region without occupying a more accurate one. The useful engineering implication is not “smaller is always better,” but that every storage and processing operation should demonstrate a marginal benefit over raw-evidence retrieval under a comparable budget.

Figure 2 repeats the cost comparison on MINT-Eval. Absolute accuracy is much lower because order and evolving state are harder than isolated fact recall, yet the same pattern appears: several memory systems consume additional tokens while remaining below RAG. The paper reports one-round and two-round RAG separately; the per-query benefit and tax analysis uses one-round RAG as the reference so that the broader two-round budget does not itself become the explanation for every observed difference.

Where Benefit and Tax Come From

The Venn plots make query migration visible. On LongMemEval-s with Qwen3.6-27B, A-Mem has a positive net of nine questions but still contains 72 questions that both systems miss, along with losses in both directions. SimpleMem has a net loss of 20. A positive subsystem gain is therefore not enough: a memory may create useful new successes while losing more of the successes that the original interface already provided.

MINT-Eval produces larger movement. PlugMem has many memory-only and RAG-only successes, but the figure reports a positive net of 123, showing that it materially changes the answerable query set rather than reproducing RAG. A-Mem, SimpleMem, and Memory-R2 show the opposite imbalance. This is why the paper’s method is more informative than a leaderboard: the figure directly separates new capability from lost capability.

The first mechanism behind the tax is faithfulness. A write pipeline that summarizes, compresses, extracts, or rewrites may remove a necessary qualifier, collapse conflicting observations, or alter temporal reference. If evidence disappears from the constructed store, no later retriever can recover it. The authors map benchmark-provided gold raw turns to gold memory units, then replay answering with each evidence type separately. The comparison isolates information lost during construction from retrieval failure.

Table 2 shows the sampled severity of this loss. MemoryOS preserves 98.4 percent of sampled facts without loss but has 1.6 percent with no corresponding unit. OpenViking has only 39.4 percent no-loss cases, 9.6 percent partial loss, 13.0 percent major loss, and 38.0 percent no unit. The exact sampled rates should not be generalized to every fact, but the spread across systems establishes that successful ingestion is not equivalent to faithful preservation.

The second mechanism is accessibility. Even when evidence survives, it may be fragmented, mixed with unrelated details, or paraphrased into a form that no longer matches the user query. The paper compares query-to-raw evidence similarity $\mathrm{sim}(q,g_{\mathrm{raw}})$ with query-to-memory-unit similarity $\mathrm{sim}(q,g_{\mathrm{mem}})$. Because each system uses its own store, absolute cross-system similarities are only indicative; the meaningful signal is how construction shifts the retrieval alignment of the same evidence.

Figure 6 adds a third mechanism: robustness. When the construction model is fixed and the answering model changes, many memory systems lose more accuracy than RAG. When the answering model is fixed and the construction model changes, memory content itself shifts because ingestion depends on model judgment. A deployed agent may write with one model and later answer with another, so this is not an artificial configuration. Extra model-dependent processing creates an additional failure surface that persists as a store of accumulated decisions.

Benchmark Headroom Changes How Visible the Tax Becomes

The evaluation setting also controls how much memory can contribute. If most questions can be solved by matching a distinctive term to a short history segment, heavy abstraction has little room to add value while still carrying risk. The authors compare no-memory, IDF keyword retrieval, dense RAG, and hybrid RAG across many memory benchmarks. On LongMemEval-s, IDF keyword reaches 46.2 and hybrid RAG reaches 77.6. The corresponding scores are 39.5 and 46.8 on PersonaMem-v2, and 20.4 and 28.6 on MINT-Eval.

This finding is not an excuse for poor memory systems. It is a requirement on benchmark design. A test intended to measure long-term memory should report a well-configured raw retrieval baseline so readers can see the available headroom. Otherwise, a small loss introduced by construction can dominate a small potential gain, while a benchmark with many easy queries can make lightweight approaches appear artificially strong simply because retrieval already does most of the work.

The authors also test whether RAG’s strength depends on one lucky configuration. They vary buffer size, turn-level chunking, BM25 versus dense retrieval, embedding models, and rerankers. Most variants remain within a few points, and the overall conclusion remains stable. The strongest configuration is not universal, but a published hybrid retrieval pipeline is consistently competitive. The methodological standard is therefore reproducibility and reasonable tuning, not the claim that one fixed RAG recipe is optimal for every agent.

Beyond Conversational Memory

The same framework applies to experiential memory, where an agent distills prior trajectories into workflows, strategies, or skills. Evaluations cover ALFWorld, WebShop, Mind2Web, BigCodeBench, AppWorld, and tool-use tasks. Web tasks sometimes reward trace distillation because reusable workflows can outperform retrieval of entire past trajectories. Tool-use and coding settings show several memory methods below naive trajectory RAG. The tax depends on whether a task needs an abstract strategy or exact operational evidence.

For world-model memory, the authors compare raw frames, raw key-value caches, learned keys and values, visual latents, and memory tokens on long-horizon consistency. I3DM with raw frames achieves SSIM 0.560 and 1-LPIPS 0.704 on C-as-M-eval, and 0.430 and 0.476 on MemoBench. WorldKV with raw keys and values reaches 0.372 and 0.459, while MagicWorld with visual latents reaches 0.388 and 0.341. Compact learned representations do not consistently beat direct replay, so the construction-versus-fidelity trade-off also appears in temporal visual state.

The first proposed mitigation is source augmentation. Memory still provides organization and abstraction, but retrieval also draws a small number of raw-history chunks and reranks both sources together. On Qwen3.6-27B, SimpleMem rises from 73.6 to 82.8 on LongMemEval-s while its tax falls from 12.0 to 5.2. MemoryOS rises from 73.4 to 76.4, and Mem-T from 75.0 to 80.0. On PersonaMem-v2, SimpleMem and MemoryOS rise to 49.6 and 53.0.

The second mitigation is recursive memory: store past queries, the evidence they found, and retrieval paths so that similar future requests can directly reach information that originally required deep search. The gains on PersonaMem-v2 are small, such as SimpleMem moving from 46.7 to 46.8 and MemoryOS from 48.2 to 49.1. LongMemEval-s has one query per corpus, so cross-query recursion is not applicable. The authors present this as a limited proof of concept rather than a universal remedy.

Code Review: The Data Pipeline Fixes Comparable Samples

The public repository makes the evaluated subsets auditable. In common/bench/prepare.py, LongMemEval keeps all 500 questions and constructs a stratified 100-question subset with a fixed random seed. PersonaMem-v2 groups rows by chat-history link and selects 40 personas containing 934 questions. MINT-Eval selects complete rows across its four subdomains. Complete-row selection matters because questions such as “N steps back” depend on the full step sequence; truncating the corpus would change the task rather than merely reduce its size.

For MINT dialogue, the script splits session blocks on lines beginning with user: or assistant:. Each turn carries a timestamp, block_idx, and turn_idx, while a leading scenario line is retained as a separate item with role scenario. This gives all systems the same ordered units and explains why the unified prompt requires the model to count using the explicit step labels rather than the order in which retrieved units happen to be printed.

The repository also publishes one answer prompt per benchmark and a shared judge. The LongMemEval prompt defines date formatting, relative-time resolution, recency when memories conflict, and evidence combination across sessions. PersonaMem shuffles four options with a per-question seed to avoid always placing the correct answer last. MINT requires retention of step indices and gives later state updates precedence. These are not minor formatting choices; they define what evidence the model is expected to combine when all systems are compared.

Code Review: Results Are Traceable, but Full Reproduction Is Incomplete

The judge implementation in common/judge/prompts.py supports semantic equivalence for open answers, rubric-based evaluation for preferences, and a special fallback for PersonaMem responses that quote an option instead of naming its letter. The fallback explicitly rejects hedging or responses that commit to multiple options. Parsing returns 1 for correct, 0 for wrong, or None for an unparsable verdict, so parser failure can be tracked separately rather than silently converted into a negative label.

The boundary between reusable code and released evidence is explicit. The README links Hugging Face repositories containing QA outputs, judge verdicts, cost statistics, memory stores, ablations, and other intermediates. It still marks baseline reproduction code as TODO. In a current clone, most of the eleven harness/ directories contain only .gitkeep; the runnable public surface centers on benchmark download and preparation, shared prompts, judging, and result organization.

This should be described accurately in both directions. The paper’s empirical claim is unusually auditable because raw model outputs and verdicts are available, and the common evaluation contracts can be inspected. At the same time, an independent team cannot yet regenerate every main-table number from the repository with one documented path. For a paper whose contribution is a comparison methodology, publishing the full harnesses would be a high-value next step.

Limitations and Independent Assessment

The authors state that their baseline set is representative rather than exhaustive. Eleven systems cover four broad design families, but memory research is moving quickly and no study can include every new or proprietary design. They follow official configurations where possible and document controlled deviations, yet system-specific tuning could improve individual results. A single ranking should therefore be treated as evidence about this evaluation, not a permanent verdict on an architecture.

The authors also acknowledge that the main benchmarks are predominantly static, offline, and English-language. They do not model long-lived deployments with interleaved writes and reads, concurrent users, deletion requests, privacy constraints, or multilingual interaction. Compute and API costs force stratified subsets, the largest proprietary backbone is GPT-5.6 Luna low, and several experiments evaluate only a fraction of the available data. Those choices materially bound the scope of the conclusion.

From an independent review perspective, the largest measurement uncertainty is the LLM judge. The paper reports human agreement of 95.1 percent, 100 percent, and 97.3 percent across relevant checks, which is reassuring but not decisive for temporal, preference, or nuanced semantic cases. Because benefit and tax are defined from judge labels, a systematic category-specific error would alter both the absolute scores and their decomposition. Per-question human review and confidence intervals by type would strengthen the evidence.

Finally, the result is reference-dependent by design. Memory tax is defined against a particular naive RAG configuration. If the reference adds stronger query rewriting, iterative search, or a different chunking strategy, the numerical benefit and tax will change. The RAG ablations show that the overall critique is not driven by one weak baseline, but they also imply that memory systems should be benchmarked against continuously improving retrieval. The goal is a defensible marginal gain under explicit cost and robustness constraints, not victory over a deliberately simple straw man.

Conclusion

The paper’s strongest contribution is to turn agent memory into a marginal-value question. Holding broad retrieval budgets, answer prompts, backbones, and judging contracts as fixed as possible, it shows that memory can solve queries RAG misses while also deleting, fragmenting, diluting, or obscuring evidence that raw history already exposes. The reported pattern of 90 out of 99 memory results below naive RAG is not a blanket rejection of memory; it is a demand that added infrastructure demonstrate both sides of its effect.

For engineering practice, three implications follow. Every memory layer should report a raw-history retrieval baseline. Construction should be evaluated not only by write success or compression rate, but by evidence preservation, retrieval accessibility, and robustness across model changes. Keeping a lightweight path back to source history is a practical compromise: abstraction handles organization and efficiency, while raw evidence can recover completeness when the abstraction is insufficient.

For embodied agents, the lesson is not to abandon visual or experiential memory. It is to distinguish reusable summaries from state-critical evidence. Losing an object state, timestamp, action precondition, or observation can be far more consequential than losing a sentence in a conversational answer. Traceable memory units, source augmentation, and regression tests across construction and consumption models are therefore more valuable than increasing storage capacity without measuring what becomes answerable.

Key Line

The value of memory is not how much of the past it stores, but what it makes answerable beyond raw history and what it quietly makes unanswerable.