mlsanityData to 14 Oct 2026

RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation

arXiv 2608.23568 ↗First posted 5 Jun 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Memory and RAG evaluations often treat the answering model's input as an implementation detail, even though systems may render the same history as a memory entry, summary, typed record, or raw excerpt. We introduce RENDER, a benchmark control that fixes the conversation while varying the reader-facing artifact. RENDER combines a five-level packet ladder, localizing when answer-bearing content enters the input, with deterministic templates approximating ChatGPT-style entries, LangChain summaries, MemGPT-style typed records, and raw conversation. On 500 LongMemEval questions and nine models, matched-budget resolved packets beat recency-truncated raw dialogue by 42.4-72.6 points. In deployed-style templates, best-worst spread is 24.6-48.8 points per model; under the primary scorer, ChatGPT-style entries have higher point estimates than raw conversation on 7 of 9 models. Judge rescoring preserves the positive aggregate effect, but model-specific significance is mixed. Three models scoring 0 percent on formal ledger packets answer the same facts from natural-language entries at 45.4-53.4 percent. The effect persists under retrieval noise and transfers to HotpotQA, suggesting that memory/RAG evaluations should report or control the reader-facing artifact.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

46 of 57 references.

References · 46

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  12. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
34 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  11. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  12. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  13. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  14. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  15. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  16. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  17. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  18. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  19. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  20. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  21. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  22. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  23. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  24. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  25. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  26. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  27. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  28. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  29. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 58
  30. 2024Belief Revision: The Adaptability of Large Language Models Reasoningcited by 30
  31. 2025ENGRAM: Effective, Lightweight Memory Orchestration for Conversational Agentscited by 15
  32. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Modelscited by 9
  33. 2025A Systematic Study of Time Limit Exceeded Errors in Online Programming Assignmentscited by 3
  34. 2025Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings
  2. 2025Convomem Benchmark: Why Your First 150 Conversations Don't Need RAG
  3. 2026Zero-Mem: Zero-Token Memory Operations for LLM Agents
  4. 2026SodaMem: Evidence-Grounded Temporal Graph Memory for LLM Agents
  5. 2026When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents
  6. 2026MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
  7. 2026LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture
  8. 2026CMT-RAG: Complementary Memory Traces for Multi-turn Multi-hop RAG