mlsanityData to 7 Oct 2026

Large Language Models Are Semi-Parametric Reinforcement Learning Agents

arXiv 2306.07929 ↗First posted 9 Jun 2023cs.CL cs.AIcited by 42

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Inspired by the insights in cognitive science with respect to human memory and reasoning mechanism, a novel evolvable LLM-based (Large Language Model) agent framework is proposed as REMEMBERER. By equipping the LLM with a long-term experience memory, REMEMBERER is capable of exploiting the experiences from the past episodes even for different task goals, which excels an LLM-based agent with fixed exemplars or equipped with a transient working memory. We further introduce Reinforcement Learning with Experience Memory (RLEM) to update the memory. Thus, the whole system can learn from the experiences of both success and failure, and evolve its capability without fine-tuning the parameters of the LLM. In this way, the proposed REMEMBERER constitutes a semi-parametric RL agent. Extensive experiments are conducted on two RL task sets to evaluate the proposed framework. The average results with different initialization and training sets exceed the prior SOTA by 4% and 2% for the success rate on two task sets and demonstrate the superiority and robustness of REMEMBERER.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 27 of its 38 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

27 of 38 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
15 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  4. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  5. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  6. 2023Language Models can Solve Computer Taskscited by 371
  7. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  8. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  9. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  10. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  11. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  12. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  13. 2023RET-LLM: Towards a General Read-Write Memory for Large Language Modelscited by 64
  14. 2023Memory Augmented Large Language Models are Computationally Universalcited by 45
  15. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 14; 2026: 13.

42 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  3. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  4. 2023CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalizationcited by 88
  5. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  6. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 49
  7. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  8. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agentscited by 41
  9. 2024Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacycited by 36
  10. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  11. 2023Temporal Knowledge Question Answering via Abstract Reasoning Inductioncited by 23
  12. 2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
30 more
  1. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 17
  2. 2024Leveraging Print Debugging to Improve Code Generation in Large Language Modelscited by 15
  3. 2024Agent Alignment in Evolving Social Normscited by 14
  4. 2024Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learningcited by 13
  5. 2025TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learningcited by 13
  6. 2025Truly Self-Improving Agents Require Intrinsic Metacognitive Learningcited by 13
  7. 2024Can LLM be a Good Path Planner based on Prompt Engineering? Mitigating the Hallucination for Path Planningcited by 11
  8. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  9. 2026Auto-Dreamer: Learning Offline Memory Consolidation for Language Agentscited by 9
  10. 2024Language Agents Meet Causality -- Bridging LLMs and Causal World Modelscited by 8
  11. 2026InfMem: Learning System-2 Memory Control for Long-Context Agentcited by 8
  12. 2025Enhancing Decision-Making of Large Language Models via Actor-Criticcited by 7
  13. 2025Graph-O1 : Monte Carlo Tree Search with Reinforcement Learning for Text-Attributed Graph Reasoningcited by 7
  14. 2025Digital Player: Evaluating Large Language Models based Human-like Agent in Gamescited by 6
  15. 2025Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Modelscited by 6
  16. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 5
  17. 2024Policy Learning with a Language Bottleneckcited by 4
  18. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  19. 2026MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGscited by 4
  20. 2026Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agentscited by 3
  21. 2025EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environmentscited by 2
  22. 2025Making Large Language Models Better Reasoners with Orchestrated Streaming Experiencescited by 1
  23. 2026Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QAcited by 1
  24. 2025Prompt reinforcing for long-term planning of large language modelscited by 0
  25. 2026Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Banditscited by 0
  26. 2026CSR: Infinite-Horizon Real-Time Policies with Massive Cached State Representationscited by 0
  27. 2026PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agentscited by 0
  28. 2026ASH: Agents that Self-Hone in Long-Horizon Worldscited by 0
  29. 2026Learning Stateful Predictive Knowledge From Experiencecited by 0
  30. 2026AutoFyn Technical Report: Non-Parametric Expert Iteration for Long-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents
  2. 2026AdMem: Advanced Memory for Task-solving Agents
  3. 2026Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework
  4. 2025From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory
  5. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
  6. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
  7. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics
  8. 2024A Survey on the Memory Mechanism of Large Language Model based Agents