mlsanityData to 14 Oct 2026

Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLM

arXiv 2505.10861 ↗First posted 16 May 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We investigate the usage of Large Language Model (LLM) in collecting high-quality data to warm-start Reinforcement Learning (RL) algorithms for learning in some classical Markov Decision Process (MDP) environments. In this work, we focus on using LLM to generate an off-policy dataset that sufficiently covers state-actions visited by optimal policies, then later using an RL algorithm to explore the environment and improve the policy suggested by the LLM. Our algorithm, LORO, can both converge to an optimal policy and have a high sample efficiency thanks to the LLM's good starting policy. On multiple OpenAI Gym environments, such as CartPole and Pendulum, we empirically demonstrate that LORO outperforms baseline algorithms such as pure LLM-based policies, pure RL, and a naive combination of the two, achieving up to 4 × the cumulative rewards of the pure RL baseline.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

44 of 60 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
32 more
  1. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  2. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  3. 2022Visual Language Maps for Robot Navigationcited by 365
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  5. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  6. 2022Online Decision Transformercited by 196
  7. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  8. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  9. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  10. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  11. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  12. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  13. 2019Unsupervised State Representation Learning in Ataricited by 144
  14. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  15. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
  16. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 119
  17. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  18. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  19. 2018Kickstarting Deep Reinforcement Learningcited by 81
  20. 2024Can large language models explore in-context?cited by 76
  21. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  22. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  23. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 52
  24. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  25. 2024True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learningcited by 36
  26. 2024Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programmingcited by 35
  27. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  28. 2024Grounding LLMs For Robot Task Planning Using Closed-loop State Feedbackcited by 25
  29. 2025Toward Efficient Exploration by Large Language Model Agentscited by 17
  30. 2024Efficient Sequential Decision Making with Large Language Modelscited by 16
  31. 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
  32. 2025It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Educationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?cited by 6
  2. 2026ProDVI: Programmatic Dynamics Priors for Value Network Initializationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  2. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods
  3. 2024Efficient Reinforcement Learning with Large Language Model Priors
  4. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models
  5. 2024How Can LLM Guide RL? A Value-Based Approach
  6. 2025RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models
  7. 2025ShiQ: Bringing back Bellman to LLMs
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs