mlsanityData to 7 Oct 2026

Semantic Exploration from Language Abstractions and Pretrained Representations

arXiv 2204.05080 ↗First posted 8 Apr 2022cs.LG cs.AIcited by 51

Not on the map. Why?

Abstract (arXiv)

Effective exploration is a challenge in reinforcement learning (RL). Novelty-based exploration methods can suffer in high-dimensional state spaces, such as continuous partially-observable 3D environments. We address this challenge by defining novelty using semantically meaningful state abstractions, which can be found in learned representations shaped by natural language. In particular, we evaluate vision-language representations, pretrained on natural image captioning datasets. We show that these pretrained representations drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by considering the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach in two very different task domains -- one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 8 Apr 2022.

Builds on

41 of 62 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  11. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  12. 2020Planning to Explore via Self-Supervised World Modelscited by 365
29 more
  1. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  2. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  3. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  4. 2019Self-Supervised Exploration via Disagreementcited by 263
  5. 2020First return, then explorecited by 256
  6. 2022Masked Visual Pre-training for Motor Controlcited by 246
  7. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  8. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  9. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  10. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  11. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  12. 2018Count-Based Exploration with the Successor Representationcited by 112
  13. 2018Episodic Curiosity through Reachabilitycited by 109
  14. 2018Model-Based Active Explorationcited by 105
  15. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  16. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  17. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  18. 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
  19. 2020Imitating Interactive Intelligencecited by 55
  20. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  21. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 48
  22. 2021Curious Representation Learning for Embodied Intelligencecited by 43
  23. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  24. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  25. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
  26. 2021Learning more skills through optimistic explorationcited by 37
  27. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  28. 2020Temporal Difference Uncertainties as a Signal for Explorationcited by 15
  29. 2019Language is Power: Representing States Using Natural Language in Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 19; 2024: 11; 2025: 3; 2026: 4.

51 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  3. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  4. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  5. 2023Training Language Models with Language Feedback at Scalecited by 125
  6. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  7. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  8. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  9. 2023Learning to Model the World with Languagecited by 66
  10. 2023Vision-Language Models as a Source of Rewardscited by 65
  11. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  12. 2022Training Language Models with Language Feedbackcited by 48
38 more
  1. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  2. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  3. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  4. 2022Large Language Models can Implement Policy Iterationcited by 31
  5. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  6. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  7. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  8. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  9. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  10. 2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 18
  11. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  12. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  13. 2022Language and Culture Internalisation for Human-Like Autotelic AIcited by 14
  14. 2022TextWorldExpress: Simulating Text Games at One Million Steps Per Secondcited by 12
  15. 2022Improving Policy Learning via Language Dynamics Distillationcited by 9
  16. 2022Foundation Models for Semantic Novelty in Reinforcement Learningcited by 9
  17. 2023Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actorscited by 9
  18. 2023Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learningcited by 8
  19. 2022On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learningcited by 7
  20. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 7
  21. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 7
  22. 2023DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physicscited by 6
  23. 2023Architext: Language-Driven Generative Architecture Designcited by 5
  24. 2022What is Wrong with Language Models that Can Not Tell a Story?cited by 4
  25. 2024Policy Learning with a Language Bottleneckcited by 4
  26. 2024Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Modelscited by 4
  27. 2024Visual Grounding for Object-Level Generalization in Reinforcement Learningcited by 4
  28. 2025Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agentscited by 3
  29. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  30. 2024Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstractioncited by 2
  31. 2023Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural languagecited by 1
  32. 2025Expedition & Expansion: Leveraging Semantic Representations for Goal-Directed Exploration in Continuous Cellular Automatacited by 1
  33. 2022Automatic Exploration of Textual Environments with Language-Conditioned Autotelic Agentscited by 0
  34. 2023A Song of Ice and Fire: Analyzing Textual Autotelic Agents in ScienceWorldcited by 0
  35. 2024ET tu, CLIP? Addressing Common Object Errors for Unseen Environmentscited by 0
  36. 2024From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Modelscited by 0
  37. 2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0
  38. 2026LLM-assisted Semantic Option Discovery for Facilitating Adaptive Deep Reinforcement Learningcited by 0

50 most cited of 51 All 51 (JSON)