# Understanding Early Word Learning in Situated Artificial Agents

arXiv: https://arxiv.org/abs/1710.09867 · HTML: https://mlsanity.com/paper/1710.09867/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2017.

## Built on by (most cited first, 8 of 8)

- [PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav](https://mlsanity.com/paper/2301.07302/) (2023)
- [Chess as a Testbed for Language Model State Tracking](https://mlsanity.com/paper/2102.13249/) (2021)
- [ReaSCAN: Compositional Reasoning in Language Grounding](https://mlsanity.com/paper/2109.08994/) (2021)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Learning to Mediate Disparities Towards Pragmatic Communication](https://mlsanity.com/paper/2203.13685/) (2022)
- [Learning the meanings of function words from grounded language using a visual question answering model](https://mlsanity.com/paper/2308.08628/) (2023)
- [The Construction of Reality in an AI: A Review](https://mlsanity.com/paper/2302.05448/) (2023)
- [Grounding Complex Navigational Instructions Using Scene Graphs](https://mlsanity.com/paper/2106.01607/) (2021)

## Abstract (arXiv)

Neural network-based systems can now learn to locate the referents of words and phrases in images, answer questions about visual scenes, and execute symbolic instructions as first-person actors in partially-observable worlds. To achieve this so-called grounded language learning, models must overcome challenges that infants face when learning their first words. While it is notable that models with no meaningful prior knowledge overcome these obstacles, researchers currently lack a clear understanding of how they do so, a problem that we attempt to address in this paper. For maximum control and generality, we focus on a simple neural network-based language learning agent, trained via policy-gradient methods, which can interpret single-word instructions in a simulated 3D world. Whilst the goal is not to explicitly model infant word learning, we take inspiration from experimental paradigms in developmental psychology and apply some of these to the artificial agent, exploring the conditions under which established human biases and learning effects emerge. We further propose a novel method for visualising semantic representations in the agent.