# Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration

arXiv: https://arxiv.org/abs/2002.09253 · HTML: https://mlsanity.com/paper/2002.09253/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 30 of 68)

- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance](https://mlsanity.com/paper/2310.10021/) (2023)
- [Scaling Instructable Agents Across Many Simulated Worlds](https://mlsanity.com/paper/2404.10179/) (2024)
- [OMNI: Open-endedness via Models of human Notions of Interestingness](https://mlsanity.com/paper/2306.01711/) (2023)
- [Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks](https://mlsanity.com/paper/2405.01534/) (2024)
- [Semantic Exploration from Language Abstractions and Pretrained Representations](https://mlsanity.com/paper/2204.05080/) (2022)
- [Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents](https://mlsanity.com/paper/2412.13194/) (2024)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [ELLA: Exploration through Learned Language Abstraction](https://mlsanity.com/paper/2103.05825/) (2021)
- [Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangement](https://mlsanity.com/paper/2304.14391/) (2023)
- [Augmenting Autotelic Agents with Large Language Models](https://mlsanity.com/paper/2305.12487/) (2023)
- [Interactive Learning from Activity Description](https://mlsanity.com/paper/2102.07024/) (2021)
- [SPRINT: Scalable Policy Pre-Training via Language Instruction Relabeling](https://mlsanity.com/paper/2306.11886/) (2023)
- [EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL](https://mlsanity.com/paper/2206.09674/) (2022)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach](https://mlsanity.com/paper/2305.18427/) (2023)
- [Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/2104.05043/) (2021)
- [Self-supervised Reinforcement Learning with Independently Controllable Subgoals](https://mlsanity.com/paper/2109.04150/) (2021)
- [AI Ethics Statements -- Analysis and lessons learnt from NeurIPS Broader Impact Statements](https://mlsanity.com/paper/2111.01705/) (2021)
- [Language and Culture Internalisation for Human-Like Autotelic AI](https://mlsanity.com/paper/2206.01134/) (2022)
- [Goals as Reward-Producing Programs](https://mlsanity.com/paper/2405.13242/) (2024)
- [TextWorldExpress: Simulating Text Games at One Million Steps Per Second](https://mlsanity.com/paper/2208.01174/) (2022)
- [A Framework for Learning to Request Rich and Contextually Useful Information from Humans](https://mlsanity.com/paper/2110.08258/) (2021)
- [WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-making](https://mlsanity.com/paper/2506.06725/) (2025)
- [Perceiving the World: Question-guided Reinforcement Learning for Text-based Games](https://mlsanity.com/paper/2204.09597/) (2022)
- [Discovering Sensorimotor Agency in Cellular Automata using Diversity Search](https://mlsanity.com/paper/2402.10236/) (2024)
- [SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models](https://mlsanity.com/paper/2503.01584/) (2025)

## Abstract (arXiv)

Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play. We introduce IMAGINE, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.