# Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural language

arXiv: https://arxiv.org/abs/2309.11753 · HTML: https://mlsanity.com/paper/2309.11753/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 30 of 31)

- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)
- [RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments](https://mlsanity.com/paper/2002.12292/) (2020)
- [Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation](https://mlsanity.com/paper/2109.01115/) (2021)
- [Using Natural Language for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/1903.02020/) (2019)
- [Grounded Language Learning in a Simulated 3D World](https://mlsanity.com/paper/1706.06551/) (2017)
- [Learning with AMIGo: Adversarially Motivated Intrinsic Goals](https://mlsanity.com/paper/2006.12122/) (2020)
- [Learning to Understand Goal Specifications by Modelling Reward](https://mlsanity.com/paper/1806.01946/) (2018)
- [Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration](https://mlsanity.com/paper/2002.09253/) (2020)
- [From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following](https://mlsanity.com/paper/1902.07742/) (2019)
- [Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning](https://mlsanity.com/paper/2005.07064/) (2020)
- [Semantic Exploration from Language Abstractions and Pretrained Representations](https://mlsanity.com/paper/2204.05080/) (2022)
- [Inferring Rewards from Language in Context](https://mlsanity.com/paper/2204.02515/) (2022)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.07393/) (2021)
- [ELLA: Exploration through Learned Language Abstraction](https://mlsanity.com/paper/2103.05825/) (2021)
- [Learning Rewards from Linguistic Feedback](https://mlsanity.com/paper/2009.14715/) (2020)
- [RTFM: Generalising to Novel Environment Dynamics via Reading](https://mlsanity.com/paper/1910.08210/) (2019)
- [PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards](https://mlsanity.com/paper/2007.15543/) (2020)
- [EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL](https://mlsanity.com/paper/2206.09674/) (2022)
- [Interactive Grounded Language Acquisition and Generalization in a 2D World](https://mlsanity.com/paper/1802.01433/) (2018)
- [ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1902.04546/) (2019)
- [A Narration-based Reward Shaping Approach using Grounded Natural Language Commands](https://mlsanity.com/paper/1911.00497/) (2019)
- [Dynamic population-based meta-learning for multi-agent communication with natural language](https://mlsanity.com/paper/2110.14241/) (2021)
- [Asking for Knowledge: Training RL Agents to Query External Knowledge Using Language](https://mlsanity.com/paper/2205.06111/) (2022)
- [Grounding Natural Language Commands to StarCraft II Game States for Narration-Guided Reinforcement Learning](https://mlsanity.com/paper/1906.02671/) (2019)
- [Influencing Reinforcement Learning through Natural Language Guidance](https://mlsanity.com/paper/2104.01506/) (2021)

## Built on by (most cited first, 1 of 1)

- [Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction](https://mlsanity.com/paper/2408.11816/) (2024)

## Abstract (arXiv)

Reinforcement learning is a powerful technique for learning from trial and error, but it often requires a large number of interactions to achieve good performance. In some domains, such as sparse-reward tasks, an oracle that can provide useful feedback or guidance to the agent during the learning process is really of great importance. However, querying the oracle too frequently may be costly or impractical, and the oracle may not always have a clear answer for every situation. Therefore, we propose a novel method for interacting with the oracle in a selective and efficient way, using a retrieval-based approach. We assume that the interaction can be modeled as a sequence of templated questions and answers, and that there is a large corpus of previous interactions available. We use a neural network to encode the current state of the agent and the oracle, and retrieve the most relevant question from the corpus to ask the oracle. We then use the oracle's answer to update the agent's policy and value function. We evaluate our method on an object manipulation task. We show that our method can significantly improve the efficiency of RL by reducing the number of interactions needed to reach a certain level of performance, compared to baselines that do not use the oracle or use it in a naive way.