mlsanityData to 7 Oct 2026

Improving Intrinsic Exploration with Language Abstractions

arXiv 2202.08938 ↗First posted 17 Feb 2022cs.LG cs.AI cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) agents are particularly hard to train when rewards are sparse. One common solution is to use intrinsic rewards to encourage agents to explore their environment. However, recent intrinsic exploration methods often use state-based novelty measures which reward low-level exploration and may not scale to domains requiring more abstract skills. Instead, we explore natural language as a general medium for highlighting relevant abstractions in an environment. Unlike previous work, we evaluate whether language can improve over existing exploration methods by directly extending (and comparing to) competitive intrinsic exploration baselines: AMIGo (Campero et al., 2021) and NovelD (Zhang et al., 2021). These language-based variants outperform their non-linguistic forms by 47-85% across 13 challenging tasks from the MiniGrid and MiniHack environment suites.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 55 of its 78 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

55 of 78 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
38 more
  1. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  2. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  3. 2017Count-Based Exploration with Neural Density Modelscited by 255
  4. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  5. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  6. 2020The NetHack Learning Environmentcited by 196
  7. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  8. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  9. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  10. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  11. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  12. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  13. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  14. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  15. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  16. 2018Count-Based Exploration with the Successor Representationcited by 112
  17. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  18. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  19. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  20. 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 83
  21. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  22. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  23. 2017Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learningcited by 79
  24. 2017Learning with Latent Languagecited by 75
  25. 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
  26. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  27. 2018Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learningcited by 53
  28. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 48
  29. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  30. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  31. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  32. 2019TorchBeast: A PyTorch Platform for Distributed RLcited by 41
  33. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  34. 2020Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learningcited by 32
  35. 2019Hierarchical Decision Making by Generating and Following Natural Language Instructionscited by 30
  36. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  37. 2019Automated curricula through setter-solver interactionscited by 26
  38. 2019A Narration-based Reward Shaping Approach using Grounded Natural Language Commandscited by 16

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 16; 2024: 12; 2025: 4; 2026: 3.

49 papers.

Sort:

By area:

  1. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  3. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  4. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  5. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  6. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  7. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  8. 2023Learning to Model the World with Languagecited by 66
  9. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  10. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 55
  11. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  12. 2022Exploration via Elliptical Episodic Bonusescited by 51
37 more
  1. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  2. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  3. 2022Fixing Model Bugs with Natural Language Patchescited by 32
  4. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  5. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  6. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  7. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  8. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  9. 2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 18
  10. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  11. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  12. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 16
  13. 2025Episodic Novelty Through Temporal Distancecited by 15
  14. 2022IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022cited by 14
  15. 2022Language and Culture Internalisation for Human-Like Autotelic AIcited by 14
  16. 2022TextWorldExpress: Simulating Text Games at One Million Steps Per Secondcited by 12
  17. 2023Scaling Laws for Imitation Learning in Single-Agent Gamescited by 11
  18. 2024Learning with Language-Guided State Abstractionscited by 11
  19. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 10
  20. 2022Foundation Models for Semantic Novelty in Reinforcement Learningcited by 9
  21. 2023Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actorscited by 9
  22. 2023Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Modelscited by 7
  23. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  24. 2024The Exploration-Exploitation Dilemma Revisited: An Entropy Perspectivecited by 5
  25. 2024Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Modelscited by 4
  26. 2024Visual Grounding for Object-Level Generalization in Reinforcement Learningcited by 4
  27. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  28. 2023Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teacherscited by 2
  29. 2025Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairscited by 2
  30. 2023Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural languagecited by 1
  31. 2024Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Gamecited by 1
  32. 2024Words as Beacons: Guiding RL Agents with High-Level Language Promptscited by 1
  33. 2025Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoringcited by 1
  34. 2022Automatic Exploration of Textual Environments with Language-Conditioned Autotelic Agentscited by 0
  35. 2023A Song of Ice and Fire: Analyzing Textual Autotelic Agents in ScienceWorldcited by 0
  36. 2026SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learningcited by 0
  37. 2026State Trace Rationale As Auxiliary Task in Reinforcement Learningcited by 0