# Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning

arXiv: https://arxiv.org/abs/2101.07393 · HTML: https://mlsanity.com/paper/2101.07393/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 19 Jan 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [Grounded Language Learning in a Simulated 3D World](https://mlsanity.com/paper/1706.06551/) (2017)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text](https://mlsanity.com/paper/2005.09382/) (2020)
- [Unified Pragmatic Models for Generating and Following Instructions](https://mlsanity.com/paper/1711.04987/) (2017)
- [Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation](https://mlsanity.com/paper/1906.00347/) (2019)
- [Grounded Language Learning Fast and Slow](https://mlsanity.com/paper/2009.01719/) (2020)
- [Environmental drivers of systematicity and generalization in a situated agent](https://mlsanity.com/paper/1910.00571/) (2019)
- [Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight](https://mlsanity.com/paper/1910.09664/) (2019)
- [Learning to Win by Reading Manuals in a Monte-Carlo Framework](https://mlsanity.com/paper/1401.5390/) (2014)
- [Guiding Policies with Language via Meta-Learning](https://mlsanity.com/paper/1811.07882/) (2018)
- [Representation Learning for Grounded Spatial Reasoning](https://mlsanity.com/paper/1707.03938/) (2017)
- [Alignment-based compositional semantics for instruction following](https://mlsanity.com/paper/1508.06491/) (2015)
- [Navigational Instruction Generation as Inverse Reinforcement Learning with Neural Machine Translation](https://mlsanity.com/paper/1610.03164/) (2016)

## Built on by (most cited first, 30 of 45)

- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [Improving Vision-Language-Action Model with Online Reinforcement Learning](https://mlsanity.com/paper/2501.16664/) (2025)
- [ProAgent: Building Proactive Cooperative Agents with Large Language Models](https://mlsanity.com/paper/2308.11339/) (2023)
- [Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling](https://mlsanity.com/paper/2301.12050/) (2023)
- [AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents](https://mlsanity.com/paper/2403.08978/) (2024)
- [Learning to Model the World with Language](https://mlsanity.com/paper/2308.01399/) (2023)
- [XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX](https://mlsanity.com/paper/2312.12044/) (2023)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals](https://mlsanity.com/paper/2302.04449/) (2023)
- [LLF-Bench: Benchmark for Interactive Learning from Language Feedback](https://mlsanity.com/paper/2312.06853/) (2023)
- [Leveraging Language for Accelerated Learning of Tool Manipulation](https://mlsanity.com/paper/2206.13074/) (2022)
- [Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans](https://mlsanity.com/paper/2209.13020/) (2022)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks](https://mlsanity.com/paper/2210.15629/) (2022)
- [Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication](https://mlsanity.com/paper/2409.17348/) (2024)
- [Zero-shot Task Adaptation using Natural Language](https://mlsanity.com/paper/2106.02972/) (2021)
- [Language-Guided World Models: A Model-Based Approach to AI Control](https://mlsanity.com/paper/2402.01695/) (2024)
- [SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark](https://mlsanity.com/paper/2110.10661/) (2021)
- [SPRING: Studying the Paper and Reasoning to Play Games](https://mlsanity.com/paper/2305.15486/) (2023)
- [AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback](https://mlsanity.com/paper/2309.17176/) (2023)
- [Improving Policy Learning via Language Dynamics Distillation](https://mlsanity.com/paper/2210.00066/) (2022)
- [Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards](https://mlsanity.com/paper/2301.10095/) (2023)
- [Conceptual Reinforcement Learning for Language-Conditioned Tasks](https://mlsanity.com/paper/2303.05069/) (2023)
- [Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.10865/) (2023)
- [Improving Long-Horizon Imitation Through Instruction Prediction](https://mlsanity.com/paper/2306.12554/) (2023)
- [Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use](https://mlsanity.com/paper/2410.24218/) (2024)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)

## Abstract (arXiv)

We investigate the use of natural language to drive the generalization of control policies and introduce the new multi-task environment Messenger with free-form text manuals describing the environment dynamics. Unlike previous work, Messenger does not assume prior knowledge connecting text and state observations - the control policy must simultaneously ground the game manual to entity symbols and dynamics in the environment. We develop a new model, EMMA (Entity Mapper with Multi-modal Attention) which uses an entity-conditioned attention module that allows for selective focus over relevant descriptions in the manual for each entity in the environment. EMMA is end-to-end differentiable and learns a latent grounding of entities and dynamics from text to observations using only environment rewards. EMMA achieves successful zero-shot generalization to unseen games with new dynamics, obtaining a 40% higher win rate compared to multiple baselines. However, win rate on the hardest stage of Messenger remains low (10%), demonstrating the need for additional work in this direction.