mlsanityData to 7 Oct 2026

MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning

arXiv 2511.02805 ↗First posted 4 Nov 2025cs.CL cs.AIcited by 26

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

LLM-based search agents often concatenate the full interaction history into the context, producing long and noisy inputs, and increasing compute cost and GPU memory overhead. To address this issue, we propose MemSearcher, an agent framework that maintains a compact memory during multi-turn interactions, retaining only question-relevant information and thereby keeping the context length stable across turns. Training MemSearcher is challenging because each trajectory spans multiple turns under different LLM contexts, making each turn an independent optimization target in reinforcement learning. We introduce multi-context GRPO, which propagates trajectory-level advantages to all turns for end-to-end optimization. Experiments demonstrate that MemSearcher outperforms strong history-concatenation (ReAct-style) baselines on a range of public datasets while maintaining nearly constant token counts across multi-turn interactions. The code and models will be publicly available at https://github.com/icip-cas/MemSearcher

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 65 of its 74 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

65 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agent3 Jul
  2. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent3 Jul
  3. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents18 Jun

Other references · 62

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  6. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  9. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  10. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  12. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  13. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  14. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  15. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  16. 2023Active Retrieval Augmented Generationcited by 506
  17. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  18. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  19. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 372
  20. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  21. 2024Measuring short-form factuality in large language modelscited by 345
  22. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  23. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  24. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  25. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  26. 2024Reinforcement learningcited by 236
  27. 2023Large Language Models in Finance: A Surveycited by 223
  28. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  29. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  30. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  31. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  32. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  33. 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 118
  34. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  35. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 89
  36. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  37. 2025SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesiscited by 55
  38. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 51

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 25.

26 papers.

Sort:

By area:

  1. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  2. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  3. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  4. 2026Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitationscited by 20
  5. 2026Mem-T: Densifying Rewards for Long-Horizon Memory Agentscited by 14
  6. 2026DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generationcited by 14
  7. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 13
  8. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  9. 2026Beyond Monolithic Architectures: A Multi-Agent Search and Knowledge Optimization Framework for Agentic Searchcited by 11
  10. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learningcited by 4
  11. 2026MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGscited by 4
  12. 2026AutoMem: Automated Learning of Memory as a Cognitive Skillcited by 4
14 more
  1. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  2. 2026Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agentscited by 3
  3. 2026Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memorycited by 2
  4. 2026Plan Before Search: Search Agents Need Plancited by 1
  5. 2026AWM: Answerable Working Memory for Long-Document VQA Agentscited by 1
  6. 2026Dense Process Supervision for Search Agents via Fact Utility Estimationcited by 1
  7. 2026MemTrain: Self-Supervised Context Memory Trainingcited by 0
  8. 2026R2-Searcher: Calibrating Retrieval and Reasoning Boundaries for Agentic Searchcited by 0
  9. 2026Multi-Head Recurrent Memory Agentscited by 0
  10. 2026Mint-Agent: Introducing Finance-Native Agentic Foundation Modelscited by 0
  11. 2026APEx: Distillation of Agent Procedural Experience for Adaptive Deep Research Question Answeringcited by 0
  12. 2026Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agentscited by 0
  13. 2026Remember Before You're Asked: MemDream for Self-Probing Memory Evolutioncited by 0
  14. 2026LatentHarness: Learning Latent Actions for Memory and Reasoning via Counterfactual Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning
  2. 2026From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory
  3. 2026MemFactory: Unified Inference & Training Framework for Agent Memory
  4. 2026MemGym: a Long-Horizon Memory Environment for LLM Agents
  5. 2026Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory
  6. 2026MemRetriever: Learning to Search, Reflect, and Retrieve from Long-Term Memory ↗
  7. 2026Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management
  8. 2026MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents