mlsanityData to 7 Oct 2026

Memory-Driven Self-Improvement for Decision Making with Large Language Models

arXiv 2509.26340 ↗First posted 30 Sep 2025cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Large language models (LLMs) have emerged as effective action policies for sequential decision-making (SDM) tasks due to their extensive prior knowledge. However, this broad yet general knowledge is often insufficient for specific decision-making tasks with limited task-related data, making it challenging to efficiently adapt LLMs to specific SDM tasks. To address this challenge, we propose a memory-driven self-improvement framework that combines LLM general prior knowledge with a compact memory of domain-specific experiences. Memory retains past interactions and associated Q-values, thereby capturing decision-relevant knowledge that facilitates accurate value estimation and informs the LLM prior refinement. The refined LLM prior, in turn, generates higher-reward trajectories that further enrich memory, forming a natural self-improvement framework where memory and LLM prior mutually reinforce each other. Experiments show that our memory-driven approach significantly outperforms both traditional RL and LLM-based baselines, e.g., improving performance by over 40% on in-distribution tasks and over 75% when generalized to unseen tasks in ALFWorld.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 36 of its 45 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

36 of 45 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  12. 2023Reasoning with Language Model is Planning with World Modelcited by 846
24 more
  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  2. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  3. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  4. 2018Maximum a Posteriori Policy Optimisationcited by 318
  5. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  6. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  7. 2023Reward Design with Language Modelscited by 257
  8. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  9. 2017Neural Episodic Controlcited by 148
  10. 2024LLM Critics Help Catch LLM Bugscited by 138
  11. 2016Model-Free Episodic Controlcited by 100
  12. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  13. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  14. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approachcited by 79
  15. 2024CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answeringcited by 70
  16. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoningcited by 42
  17. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  18. 2022Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Roboticscited by 23
  19. 2024Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolfcited by 22
  20. 2018Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learningcited by 21
  21. 2024Q-Probe: A Lightweight Approach to Reward Maximization for Language Modelscited by 15
  22. 2024Reflect-RL: Two-Player Online RL Fine-Tuning for LMscited by 13
  23. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
  24. 2023Neural Episodic Control with State Abstractioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
  2. 2025Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach
  3. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Making
  4. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks
  5. 2025Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models
  6. 2026S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
  7. 2025From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory
  8. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training