mlsanityData to 7 Oct 2026

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

arXiv 2311.13373 ↗First posted 22 Nov 2023cs.AIcited by 30

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recent studies have uncovered the potential of Large Language Models (LLMs) in addressing complex sequential decision-making tasks through the provision of high-level instructions. However, LLM-based agents lack specialization in tackling specific target problems, particularly in real-time dynamic environments. Additionally, deploying an LLM-based agent in practical scenarios can be both costly and time-consuming. On the other hand, reinforcement learning (RL) approaches train agents that specialize in the target task but often suffer from low sampling efficiency and high exploration costs. In this paper, we introduce a novel framework that addresses these challenges by training a smaller, specialized student RL agent using instructions from an LLM-based teacher agent. By incorporating the guidance from the teacher agent, the student agent can distill the prior knowledge of the LLM into its own model. Consequently, the student agent can be trained with significantly less data. Moreover, through further training with environment feedback, the student agent surpasses the capabilities of its teacher for completing the target task. We conducted experiments on challenging MiniGrid and Habitat environments, specifically designed for embodied AI research, to evaluate the effectiveness of our framework. The results clearly demonstrate that our approach achieves superior performance compared to strong baseline methods. Our code is available at https://github.com/ZJLAB-AMMI/LLM4Teach.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 48 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

36 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach6 Jun
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
21 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  4. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  5. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  6. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  7. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  8. 2023Reward Design with Language Modelscited by 257
  9. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  10. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  11. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  12. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  13. 2022Jump-Start Reinforcement Learningcited by 102
  14. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  15. 2018Kickstarting Deep Reinforcement Learningcited by 75
  16. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  17. 2023Augmenting Autotelic Agents with Large Language Modelscited by 29
  18. 2023Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actorscited by 9
  19. 2022Hierarchical Kickstarting for Skill Transfer in Reinforcement Learningcited by 7
  20. 2023Robot Task Planning Based on Large Language Model Representing Knowledge with Directed Graph Structurescited by 7
  21. 2023Tell Me Where to Go: A Composable Framework for Context-Aware Embodied Robot Navigationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 15; 2026: 8.

30 papers.

Sort:

By area:

  1. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  2. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  3. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  4. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  5. 2025Adaptive Graph Pruning for Multi-Agent Communicationcited by 28
  6. 2024Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learningcited by 13
  7. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  8. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  9. 2026MetaGen: Self-Evolving Roles and Topologies for Multi-Agent LLM Reasoningcited by 8
  10. 2024Efficient and Generalized end-to-end Autonomous Driving System with Latent Deep Reinforcement Learning and Demonstrationscited by 7
  11. 2025Knowledge Integration for Physics-informed Symbolic Regression Using Pre-trained Large Language Modelscited by 7
  12. 2025Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Makingcited by 5
18 more
  1. 2025Scalable UAV Multi-Hop Networking via Multi-Agent Reinforcement Learning with Large Language Modelscited by 4
  2. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Modelscited by 4
  3. 2025CoderAgent: Simulating Student Behavior for Personalized Programming Learning with Large Language Modelscited by 4
  4. 2025Large Language Model Integration with Reinforcement Learning to Augment Decision-Making in Autonomous Cyber Operationscited by 3
  5. 2025Sample-Efficient Behavior Cloning Using General Domain Knowledgecited by 2
  6. 2025Leveraging LLMs for reward function design in reinforcement learning control taskscited by 2
  7. 2026KGLAMP: Knowledge Graph-guided Language model for Adaptive Multi-robot Planning and Replanningcited by 2
  8. 2024A Survey on the Integration of Generative AI for Critical Thinking in Mobile Networkscited by 1
  9. 2026LATS: Large Language Model Assisted Teacher-Student Framework for Multi-Agent Reinforcement Learning in Traffic Signal Controlcited by 1
  10. 2026From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Modelscited by 1
  11. 2025Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme Onecited by 0
  12. 2025Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systemscited by 0
  13. 2025Empowering LLMs with Parameterized Skills for Adversarial Long-Horizon Planningcited by 0
  14. 2025SCOPE: Language Models as One-Time Teacher for Hierarchical Planning in Text Environmentscited by 0
  15. 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0
  16. 2026Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Biascited by 0
  17. 2026Conflict-Resilient Multi-Agent Reasoning via Signed Graph Modelingcited by 0
  18. 2026Can Jev be Your Q or Policy in Reinforcement Learning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Words as Beacons: Guiding RL Agents with High-Level Language Prompts
  2. 2024Aligning Agents like Large Language Models
  3. 2024Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Planning Case Study
  4. 2024Training Agents with Weakly Supervised Feedback from Large Language Models
  5. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  6. 2023LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents
  7. 2026Policy and World Modeling Co-Training for Language Agents
  8. 2023Collaborating with language models for embodied reasoning