mlsanityData to 7 Oct 2026

Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading

arXiv 2310.05029 ↗First posted 8 Oct 2023cs.CLcited by 129

Not on the map. Why?

Cited by 36 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have advanced in large strides due to the effectiveness of the self-attention mechanism that processes and compares all tokens at once. However, this mechanism comes with a fundamental issue -- the predetermined context window is bound to be limited. Despite attempts to extend the context window through methods like extrapolating the positional embedding, using recurrence, or selectively retrieving essential parts of the long sequence, long-text understanding continues to be a challenge. We propose an alternative approach which instead treats the LLM as an interactive agent, allowing it to decide how to read the text via iterative prompting. We introduce MemWalker, a method that first processes the long context into a tree of summary nodes. Upon receiving a query, the model navigates this tree in search of relevant information, and responds once it gathers sufficient information. On long-text question answering tasks our method outperforms baseline approaches that use long context windows, recurrence, and retrieval. We show that, beyond effective reading, MemWalker enhances explainability by highlighting the reasoning steps as it interactively reads the text; pinpointing the relevant text segments related to the query.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

34 of 50 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
22 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  5. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  6. 2023Adapting Language Models to Compress Contextscited by 350
  7. 2021Efficient Attentions for Long Document Summarizationcited by 342
  8. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  9. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  10. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  11. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  12. 2022Memorizing Transformerscited by 219
  13. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  14. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  15. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  16. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  17. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  18. 2023RecurrentGPT: Interactive Generation of (Arbitrarily) Long Textcited by 84
  19. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  20. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  21. 2023Learning to Reason and Memorize with Self-Notescited by 30
  22. 2018Learning to Search in Long Documents Using Document Structurecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 50; 2025: 44; 2026: 29.

129 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  3. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  4. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  5. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  6. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  7. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  8. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  9. 2025MemOS: A Memory OS for AI Systemcited by 129
  10. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  11. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  12. 2025Recursive Language Modelscited by 94
38 more
  1. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  3. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  4. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  5. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  6. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  7. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 68
  8. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  9. 2025LLM Agents for Education: Advances and Applicationscited by 59
  10. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 51
  11. 2024Towards Lifelong Dialogue Agents via Timeline-based Memory Managementcited by 50
  12. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  13. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  14. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  15. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  16. 2024M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitionscited by 38
  17. 2025TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generationcited by 37
  18. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  19. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  20. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 33
  21. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  22. 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 31
  23. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 31
  24. 2024BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidencecited by 28
  25. 2025HopRAG: Multi-Hop Reasoning for Logic-Aware Retrieval-Augmented Generationcited by 24
  26. 2025CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehensioncited by 23
  27. 2024IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abductioncited by 20
  28. 2024LLMs in the Imaginarium: Tool Learning through Simulated Trial and Errorcited by 19
  29. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  30. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  31. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  32. 2024LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generationcited by 17
  33. 2024GeAR: Graph-enhanced Agent for Retrieval-augmented Generationcited by 17
  34. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  35. 2023TigerBot: An Open Multilingual Multitask LLMcited by 14
  36. 2024A Modular Approach for Multimodal Summarization of TV Showscited by 13
  37. 2024Towards LifeSpan Cognitive Systemscited by 13
  38. 2024In-Context Former: Lightning-fast Compressing Context for Large Language Modelcited by 12

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs
  2. 2026Zero-Mem: Zero-Token Memory Operations for LLM Agents
  3. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering
  4. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  5. 2025Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents
  6. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  7. 2026Mem++: Non-Destructive Memory for Long-Term Organizational LLM Agents
  8. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA