# ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning

arXiv: https://arxiv.org/abs/1902.04546 · HTML: https://mlsanity.com/paper/1902.04546/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2019.

## Built on by (most cited first, 17 of 17)

- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation](https://mlsanity.com/paper/2109.01115/) (2021)
- [Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models](https://mlsanity.com/paper/2211.11736/) (2022)
- [Distilling Internet-Scale Vision-Language Models into Embodied Agents](https://mlsanity.com/paper/2301.12507/) (2023)
- [Interactive Learning from Activity Description](https://mlsanity.com/paper/2102.07024/) (2021)
- [A Framework for Learning to Request Rich and Contextually Useful Information from Humans](https://mlsanity.com/paper/2110.08258/) (2021)
- [AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback](https://mlsanity.com/paper/2309.17176/) (2023)
- [Improving Long-Horizon Imitation Through Instruction Prediction](https://mlsanity.com/paper/2306.12554/) (2023)
- [A Dialogue Game for Eliciting Balanced Collaboration](https://mlsanity.com/paper/2406.08202/) (2024)
- [Visual Grounding for Object-Level Generalization in Reinforcement Learning](https://mlsanity.com/paper/2408.01942/) (2024)
- [Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2405.16854/) (2024)
- [GLIDE-RL: Grounded Language Instruction through DEmonstration in RL](https://mlsanity.com/paper/2401.02991/) (2024)
- [Trust in Language Grounding: a new AI challenge for human-robot teams](https://mlsanity.com/paper/2209.02066/) (2022)
- [Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural language](https://mlsanity.com/paper/2309.11753/) (2023)
- [ETHER: Aligning Emergent Communication for Hindsight Experience Replay](https://mlsanity.com/paper/2307.15494/) (2023)
- [Learning When to Cooperate Under Heterogeneous Goals](https://mlsanity.com/paper/2603.07253/) (2026)

## Abstract (arXiv)

Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failed experience to a successful one by relabeling the goals. Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation. We present Augmenting experienCe via TeacheR's adviCE (ACTRCE), an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation. We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn. We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons. We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, and even small amount of advice is sufficient for the agent to achieve good performance.