mlsanityData to 14 Oct 2026

Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents

arXiv 2509.23040 ↗First posted 27 Sep 2025cs.CL cs.AIcited by 38

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large language models face challenges in long-context question answering, where key evidence of a query may be dispersed across millions of tokens. Existing works equip large language models with a memory buffer that is dynamically updated via a linear document scan, also known as the "memorize while reading" methods. While this approach scales efficiently, it suffers from pruning of latent evidence, information loss through overwriting, and sparse reinforcement learning signals. To tackle these challenges, we present ReMemR1, which integrates the mechanism of memory retrieval into the memory update process, enabling the agent to selectively callback historical memories for non-linear reasoning. To further strengthen training, we propose a multi-level reward design, which combines final-answer rewards with dense, step-level signals that guide effective memory use. Together, these contributions mitigate information degradation, improve supervision, and support complex multi-hop reasoning. Extensive experiments demonstrate that ReMemR1 significantly outperforms state-of-the-art baselines on long-context question answering while incurring negligible computational overhead, validating its ability to trade marginal cost for robust long-context reasoning.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 36 of its 51 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

36 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent3 Jul
  2. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics1 May
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr
  4. 2024OpenAI o1 System Card21 Dec

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
20 more
  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  2. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  5. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  9. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  10. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  11. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  12. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 241
  13. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 237
  14. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  15. 2025Qwen2.5-1M Technical Reportcited by 164
  16. 2025MemOS: A Memory OS for AI Systemcited by 138
  17. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  18. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  19. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 50
  20. 2024Enhancing Diffusion-based Point Cloud Generation with Smoothness Constraintcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 35.

38 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2026Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agentscited by 47
  3. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  4. 2026RealMem: Benchmarking LLMs in Real-World Memory-Driven Interactioncited by 20
  5. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  6. 2026MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoningcited by 15
  7. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 14
  8. 2026SAM: State-Adaptive Memory for Long-Horizon Reasoning Agentcited by 10
  9. 2026When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoningcited by 9
  10. 2026Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoningcited by 8
  11. 2026Memory Intelligence Agentcited by 7
  12. 2026Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environmentscited by 6
26 more
  1. 2026EMBER: Efficient Memory via Budgeted Evidence Retention for Long-Horizon Agentscited by 6
  2. 2026TRUSTMEM: Learning Trustworthy Memory Consolidation for LLM Agents with Long-Term Memorycited by 6
  3. 2025Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMscited by 5
  4. 2026KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planningcited by 5
  5. 2026CoMem: Context Management with A Decoupled Long-Context Modelcited by 5
  6. 2026MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialoguecited by 3
  7. 2026MAP: A Map-then-Act Paradigm for Long-Horizon Interactive Agent Reasoningcited by 2
  8. 2026MiA-Signature: Approximating Global Activation for Long-Context Understandingcited by 1
  9. 2026Cyclical Entropy Eruption: Entropy Dynamics in Agent Reinforcement Learningcited by 1
  10. 2026InfoMem: Training Long-Context Memory Agents with Answer-Conditioned Information Gaincited by 1
  11. 2026World-Model Collapse as a Phase Transitioncited by 1
  12. 2026FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agentscited by 1
  13. 2026TransMem: Transforming Hidden States into Memory for Large Language Modelscited by 1
  14. 2026MemReread: Enhancing Agentic Long-Context Reasoning via Memory-Guided Rereadingcited by 0
  15. 2026On Problems of Implicit Context Compression for Software Engineering Agentscited by 0
  16. 2026MARQUIS: A Three-Stage Pipeline for Video Retrieval-Augmented Generationcited by 0
  17. 2026PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memorycited by 0
  18. 2026Towards a Formal Definition of Agent Memory: Basis, Span, Optimality, and the Sequential Memory Problemcited by 0
  19. 2026Mint-Agent: Introducing Finance-Native Agentic Foundation Modelscited by 0
  20. 2026PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agentscited by 0
  21. 2026PluginRSI: Recursive Improvement of Agent Harnesses with Reusable Pluginscited by 0
  22. 2026Just-In-Time Agent Memory with Runtime Agentic Researchcited by 0
  23. 2026CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memorycited by 0
  24. 2026Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learningcited by 0
  25. 2026Memory Type Varies: Empowering LLM Agents for Long-Term Memory with Diverse Strategiescited by 0
  26. 2026MemDuet: Learning Dual-Process Memory for Long-Context Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory
  2. 2025Evaluating Long-Term Memory for Long-Context Question Answering
  3. 2026EdgeMem: LLM-Free Agent Memory Construction and Retrieval via Evidence-Preserving Multi-Anchor Hypergraph
  4. 2026MemRetriever: Learning to Search, Reflect, and Retrieve from Long-Term Memory ↗
  5. 2025Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents
  6. 2026MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards
  7. 2026DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Agent Memory
  8. 2026MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agents