mlsanityData to 7 Oct 2026

ELLA: Exploration through Learned Language Abstraction

arXiv 2103.05825 ↗First posted 10 Mar 2021cs.CL cs.AI cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Building agents capable of understanding language instructions is critical to effective and robust human-AI collaboration. Recent work focuses on training these agents via reinforcement learning in environments with synthetic language; however, instructions often define long-horizon, sparse-reward tasks, and learning policies requires many episodes of experience. We introduce ELLA: Exploration through Learned Language Abstraction, a reward shaping approach geared towards boosting sample efficiency in sparse reward environments by correlating high-level instructions with simpler low-level constituents. ELLA has two key elements: 1) A termination classifier that identifies when agents complete low-level instructions, and 2) A relevance classifier that correlates low-level instructions with success on high-level tasks. We learn the termination classifier offline from pairs of instructions and terminal states. Notably, in departure from prior work in language and abstraction, we learn the relevance classifier online, without relying on an explicit decomposition of high-level instructions to low-level instructions. On a suite of complex BabyAI environments with varying instruction complexities and reward sparsity, ELLA shows gains in sample efficiency relative to language-based shaping and traditional RL methods.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

36 of 48 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  8. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  9. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  11. 2020Experience Grounds Languagecited by 241
  12. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
24 more
  1. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  2. 2019Good-Enough Compositional Data Augmentationcited by 158
  3. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  4. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  5. 2019Explaining Classifiers with Causal Concept Effect (CaCE)cited by 135
  6. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  7. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  8. 2016Data Recombination for Neural Semantic Parsingcited by 111
  9. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  10. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  11. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  12. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  13. 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
  14. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  15. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  16. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  17. 2020Learning Adaptive Language Interfaces through Decompositioncited by 22
  18. 2017Naturalizing a Programming Language via Interactive Learningcited by 20
  19. 2017Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularitiescited by 19
  20. 2019A Narration-based Reward Shaping Approach using Grounded Natural Language Commandscited by 16
  21. 2019HIGhER : Improving instruction following with Hindsight Generation for Experience Replaycited by 13
  22. 2017A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructionscited by 4
  23. 2020Useful Policy Invariant Shaping from Arbitrary Advicecited by 4
  24. 2017A Deep Compositional Framework for Human-like Language Acquisition in Virtual Environmentcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 15; 2024: 12; 2025: 3; 2026: 1.

42 papers.

Sort:

By area:

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  4. 2023Reward Design with Language Modelscited by 257
  5. 2024RT-H: Action Hierarchies Using Languagecited by 204
  6. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  7. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  8. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  9. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  10. 2023Learning to Model the World with Languagecited by 66
  11. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  12. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
30 more
  1. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  2. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  3. 2023Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Roboticscited by 35
  4. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  5. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  6. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 19
  7. 2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 18
  8. 2023State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understandingcited by 17
  9. 2022Language and Culture Internalisation for Human-Like Autotelic AIcited by 14
  10. 2022Perceiving the World: Question-guided Reinforcement Learning for Text-based Gamescited by 10
  11. 2023Toward Grounded Commonsense Reasoningcited by 9
  12. 2023LLM Augmented Hierarchical Agentscited by 9
  13. 2024Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewardscited by 9
  14. 2023Improving Long-Horizon Imitation Through Instruction Predictioncited by 8
  15. 2023A Picture is Worth a Thousand Words: Language Models Plan from Pixelscited by 6
  16. 2023Architext: Language-Driven Generative Architecture Designcited by 5
  17. 2024Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Modelscited by 4
  18. 2024Visual Grounding for Object-Level Generalization in Reinforcement Learningcited by 4
  19. 2024Automated Rewards via LLM-Generated Progress Functionscited by 4
  20. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  21. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  22. 2024XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniquescited by 2
  23. 2024Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimatorscited by 2
  24. 2025Lightweight Robust Direct Preference Optimizationcited by 2
  25. 2023Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural languagecited by 1
  26. 2023A Song of Ice and Fire: Analyzing Textual Autotelic Agents in ScienceWorldcited by 0
  27. 2024mathrmE2CFD: Towards Effective and Efficient Cost Function Design for Safe Reinforcement Learning via Large Language Modelcited by 0
  28. 2024CAREL: Instruction-guided reinforcement learning with cross-modal auxiliary objectivescited by 0
  29. 2025Inductive Biases for Zero-shot Systematic Generalization in Language-informed Reinforcement Learningcited by 0
  30. 2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0