# Reward Learning from Narrated Demonstrations

arXiv: https://arxiv.org/abs/1804.10692 · HTML: https://mlsanity.com/paper/1804.10692/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2018.

## Built on by (most cited first, 13 of 13)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following](https://mlsanity.com/paper/1902.07742/) (2019)
- [LazyDAgger: Reducing Context Switching in Interactive Imitation Learning](https://mlsanity.com/paper/2104.00053/) (2021)
- [Invariance in Policy Optimisation and Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2203.07475/) (2022)
- [Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning](https://mlsanity.com/paper/2402.17930/) (2024)
- [Inferring the Goals of Communicating Agents from Actions and Instructions](https://mlsanity.com/paper/2306.16207/) (2023)
- [RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback](https://mlsanity.com/paper/2308.04332/) (2023)
- [Reward Learning from Multiple Feedback Types](https://mlsanity.com/paper/2502.21038/) (2025)
- [A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learning](https://mlsanity.com/paper/2108.03222/) (2021)
- [Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations](https://mlsanity.com/paper/2509.24250/) (2025)
- [Neural Task Success Classifiers for Robotic Manipulation from Few Real Demonstrations](https://mlsanity.com/paper/2107.00722/) (2021)
- [Exploring and Addressing Reward Confusion in Offline Preference Learning](https://mlsanity.com/paper/2407.16025/) (2024)
- [ViSTa Dataset: Do vision-language models understand sequential tasks?](https://mlsanity.com/paper/2411.13211/) (2024)

## Abstract (arXiv)

Humans effortlessly "program" one another by communicating goals and desires in natural language. In contrast, humans program robotic behaviours by indicating desired object locations and poses to be achieved, by providing RGB images of goal configurations, or supplying a demonstration to be imitated. None of these methods generalize across environment variations, and they convey the goal in awkward technical terms. This work proposes joint learning of natural language grounding and instructable behavioural policies reinforced by perceptual detectors of natural language expressions, grounded to the sensory inputs of the robotic agent. Our supervision is narrated visual demonstrations(NVD), which are visual demonstrations paired with verbal narration (as opposed to being silent). We introduce a dataset of NVD where teachers perform activities while describing them in detail. We map the teachers' descriptions to perceptual reward detectors, and use them to train corresponding behavioural policies in simulation.We empirically show that our instructable agents (i) learn visual reward detectors using a small number of examples by exploiting hard negative mined configurations from demonstration dynamics, (ii) develop pick-and place policies using learned visual reward detectors, (iii) benefit from object-factorized state representations that mimic the syntactic structure of natural language goal expressions, and (iv) can execute behaviours that involve novel objects in novel locations at test time, instructed by natural language.