mlsanityData to 14 Oct 2026

Guiding Pretraining in Reinforcement Learning with Large Language Models

arXiv 2302.06692 ↗First posted 13 Feb 2023cs.LG cs.AI cs.CLcited by 187

Not on the map. Why?

Cited by 41 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning algorithms typically struggle in the absence of a dense, well-shaped reward function. Intrinsically motivated exploration methods address this limitation by rewarding agents for visiting novel states or transitions, but these methods offer limited benefits in large environments where most discovered novelty is irrelevant for downstream tasks. We describe a method that uses background knowledge from text corpora to shape exploration. This method, called ELLM (Exploring with LLMs) rewards an agent for achieving goals suggested by a language model prompted with a description of the agent's current state. By leveraging large-scale language model pretraining, ELLM guides agents toward human-meaningful and plausibly useful behaviors without requiring a human in the loop. We evaluate ELLM in the Crafter game environment and the Housekeep robotic simulator, showing that ELLM-trained agents have better coverage of common-sense behaviors during pretraining and usually match or improve performance on a range of downstream tasks. Code available at https://github.com/yuqingd/ellm.

Key facts. Cited by 187 papers in the map (data through 14 Oct 2026). Builds on 48 of its 71 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

48 of 71 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  10. 2016Concrete Problems in AI Safetycited by 1,642
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
36 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  2. 2018Exploration by Random Network Distillationcited by 993
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  7. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  8. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  9. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  10. 2018Human Action Recognition and Prediction: A Surveycited by 227
  11. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  12. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  13. 2021Reinforcement Learning with Prototypical Representationscited by 179
  14. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  15. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  16. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  17. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  18. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  19. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  20. 2021Skill Induction and Planning with Latent Languagecited by 99
  21. 2021A Survey of Modern Deep Learning based Object Detection Modelscited by 98
  22. 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 85
  23. 2019A survey on intrinsic motivation in reinforcement learningcited by 81
  24. 2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
  25. 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
  26. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  27. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 58
  28. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  29. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  30. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  31. 2019Environmental drivers of systematicity and generalization in a situated agentcited by 50
  32. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  33. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  34. 2018Investigating Human Priors for Playing Video Gamescited by 39
  35. 2019ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learningcited by 17
  36. 2022Learning to Generalize with Object-centric Agents in the Open World Survival Game Craftercited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 41; 2024: 71; 2025: 48; 2026: 27.

187 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  2. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  3. 2023Learning Interactive Real-World Simulatorscited by 458
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  5. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  6. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  7. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  8. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  9. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  10. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  11. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 88
  12. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
38 more
  1. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  2. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 75
  3. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 70
  4. 2023Learning to Model the World with Languagecited by 68
  5. 2023Vision-Language Models as a Source of Rewardscited by 66
  6. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agentscited by 63
  7. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 61
  8. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 59
  9. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 58
  10. 2026ASPIRE: Agentic /Skills Discovery for Roboticscited by 53
  11. 2023A Short Survey of Viewing Large Language Models in Legal Aspectcited by 52
  12. 2023Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaborationcited by 52
  13. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 52
  14. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  15. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 46
  16. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 44
  17. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 43
  18. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  19. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  20. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  21. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 31
  22. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 31
  23. 2023Augmenting Autotelic Agents with Large Language Modelscited by 30
  24. 2023Large Language Model as a Policy Teacher for Training Reinforcement Learning Agentscited by 30
  25. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 30
  26. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  27. 2025ViPRA: Video Prediction for Robot Actionscited by 26
  28. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 25
  29. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
  30. 2024A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Healthcited by 24
  31. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representationscited by 24
  32. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 23
  33. 2023Can ChatGPT Enable ITS? The Case of Mixed Traffic Control via Reinforcement Learningcited by 20
  34. 2024When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challengescited by 20
  35. 2024Motion Avatar: Generate Human and Animal Avatars with Arbitrary Motioncited by 19
  36. 2023Out of the Cage: How Stochastic Parrots Win in Cyber Security Environmentscited by 18
  37. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  38. 2023Pre-Trained Large Language Models for Industrial Controlcited by 16

50 most cited of 187 All 187 (JSON)