mlsanityData to 7 Oct 2026

Memento: Fine-tuning LLM Agents without Fine-tuning LLMs

arXiv 2508.16153 ↗First posted 22 Aug 2025cs.LG cs.CLcited by 119

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

In this paper, we introduce a novel learning paradigm for Adaptive Large Language Model (LLM) agents that eliminates the need for fine-tuning the underlying LLMs. Existing approaches are often either rigid, relying on static, handcrafted reflection workflows, or computationally intensive, requiring gradient updates of LLM model parameters. In contrast, our method enables low-cost continual adaptation via memory-based online reinforcement learning. We formalise this as a Memory-augmented Markov Decision Process (M-MDP), equipped with a neural case-selection policy to guide action decisions. Past experiences are stored in an episodic memory, either differentiable or non-parametric. The policy is continually updated based on environmental feedback through a memory rewriting mechanism, whereas policy improvement is achieved through efficient memory reading (retrieval). We instantiate our agent model in the deep research setting, namely Memento, which attains top-1 on GAIA validation (87.88% Pass@3) and 79.40% on the test set. It reaches 66.6% F1 and 80.4% PM on the DeepResearcher dataset, outperforming the state-of-the-art training-based method, while case-based memory adds 4.7% to 9.6% absolute points on out-of-distribution tasks. Our approach offers a scalable and efficient pathway for developing generalist LLM agents capable of continuous, real-time learning without gradient updates, advancing machine learning towards open-ended skill acquisition and deep research scenarios. The code is available at https://github.com/Agent-on-the-Fly/Memento.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 62 of its 82 references that are in the map. First posted on arXiv 22 Aug 2025.

Builds on

62 of 82 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving8 Jul
  2. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory28 Apr
  3. 2025MemInsight: Autonomous Memory Augmentation for LLM Agents27 Mar
  4. 2025A-MEM: Agentic Memory for LLM Agents17 Feb
  5. 2024Self-evolving Agents with reflective and memory-augmented abilities1 Sep

Other references · 57

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
38 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  5. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  9. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  10. 2025Humanity's Last Examcited by 467
  11. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  12. 2023Cognitive Architectures for Language Agentscited by 422
  13. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  14. 2024Measuring short-form factuality in large language modelscited by 345
  15. 2024Agent Workflow Memorycited by 342
  16. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  17. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  18. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  19. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  20. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  21. 2024Many-Shot In-Context Learningcited by 203
  22. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  23. 2017Neural Episodic Controlcited by 148
  24. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  25. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  26. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  27. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 119
  28. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 114
  29. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 97
  30. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  31. 2024CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answeringcited by 70
  32. 2022Recitation-Augmented Language Modelscited by 65
  33. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  34. 2025START: Self-taught Reasoner with Toolscited by 47
  35. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  36. 2021AlphaEvolve: A Learning Framework to Discover Novel Alphas in Quantitative Investmentcited by 31
  37. 2023Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoningcited by 30
  38. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 104.

119 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2026MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memorycited by 113
  3. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 68
  4. 2026How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settingscited by 62
  5. 2026Memento-Skills: Let Agents Design Agentscited by 56
  6. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 52
  7. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 51
  8. 2026SkillOS: Learning Skill Curation for Self-Evolving Agentscited by 45
  9. 2026Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agentscited by 44
  10. 2026Meta Context Engineering via Agentic Skill Evolutioncited by 42
  11. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  12. 2026HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundrycited by 37
38 more
  1. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 33
  2. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  3. 2025LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarioscited by 25
  4. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  5. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  6. 2026SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolutioncited by 18
  7. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  8. 2026Learning to Continually Learn via Meta-learning Agentic Memory Designscited by 16
  9. 2026MEMENTO: Teaching LLMs to Manage Their Own Contextcited by 16
  10. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  11. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  12. 2026Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memorycited by 11
  13. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  14. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  15. 2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
  16. 2026Combee: Scaling Prompt Learning for Self-Improving Language Model Agentscited by 8
  17. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 8
  18. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 7
  19. 2026FlashMem: Distilling Intrinsic Latent Memory via Computation Reusecited by 7
  20. 2026TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarkingcited by 7
  21. 2026Workspace Optimization: How to Train Your Agentcited by 7
  22. 2026Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agentscited by 7
  23. 2026SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probingcited by 6
  24. 2026Retrieval-Augmented LLM Agents: Learning to Learn from Experiencecited by 6
  25. 2026Memory Intelligence Agentcited by 6
  26. 2025Surfer 2: The Next Generation of Cross-Platform Computer Use Agentscited by 5
  27. 2025Dingtalk DeepResearch: A Unified Multi Agent Framework for Adaptive Intelligence in Enterprise Environmentscited by 5
  28. 2025Smarter Together: Creating Agentic Communities of Practice through Shared Experiential Learningcited by 5
  29. 2025Memento 2: Learning by Stateful Reflective Memorycited by 5
  30. 2026Large Language Model Agents Are Not Always Faithful Self-Evolverscited by 5
  31. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learningcited by 5
  32. 2026Co-Evolving Skill Generation and Policy Optimizationcited by 5
  33. 2025Adapting Like Humans: A Metacognitive Agent with Test-time Reasoningcited by 4
  34. 2025Unifying Dynamic Tool Creation and Cross-Task Experience Sharing through Cognitive Memory Architecturecited by 4
  35. 2026MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGscited by 4
  36. 2026FragFuse: Bypassing Access Control of Large Language Model Agents via Memory-Based Query Fragmentation and Fusioncited by 4
  37. 2025ResearStudio: A Human-Intervenable Framework for Building Controllable Deep-Research Agentscited by 3
  38. 2026Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesiscited by 3

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents
  2. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
  3. 2026Choosing How to Remember: Adaptive Memory Structures for LLM Agents
  4. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
  5. 2026Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
  6. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics
  7. 2026MemFactory: Unified Inference & Training Framework for Agent Memory
  8. 2025From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory