# Learning Invariable Semantical Representation from Language for Extensible Policy Generalization

arXiv: https://arxiv.org/abs/2202.00466 · HTML: https://mlsanity.com/paper/2202.00466/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 29 of 29)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)
- [Using Natural Language for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/1903.02020/) (2019)
- [Grounded Language Learning in a Simulated 3D World](https://mlsanity.com/paper/1706.06551/) (2017)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [Learning to Understand Goal Specifications by Modelling Reward](https://mlsanity.com/paper/1806.01946/) (2018)
- [Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning](https://mlsanity.com/paper/1712.07294/) (2017)
- [Language Understanding for Text-based Games Using Deep Reinforcement Learning](https://mlsanity.com/paper/1506.08941/) (2015)
- [Compositional Generalization by Learning Analytical Expressions](https://mlsanity.com/paper/2006.10627/) (2020)
- [Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences](https://mlsanity.com/paper/1506.04089/) (2015)
- [Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies](https://mlsanity.com/paper/1807.07665/) (2018)
- [Towards Deep Symbolic Reinforcement Learning](https://mlsanity.com/paper/1609.05518/) (2016)
- [Learning to Win by Reading Manuals in a Monte-Carlo Framework](https://mlsanity.com/paper/1401.5390/) (2014)
- [Counting to Explore and Generalize in Text-based Games](https://mlsanity.com/paper/1806.11525/) (2018)
- [RTFM: Generalising to Novel Environment Dynamics via Reading](https://mlsanity.com/paper/1910.08210/) (2019)
- [Hierarchical Decision Making by Generating and Following Natural Language Instructions](https://mlsanity.com/paper/1906.00744/) (2019)
- [Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control](https://mlsanity.com/paper/1910.14033/) (2019)
- [Exploiting Language Instructions for Interpretable and Compositional Reinforcement Learning](https://mlsanity.com/paper/2001.04418/) (2020)

## Abstract (arXiv)

Recently, incorporating natural language instructions into reinforcement learning (RL) to learn semantically meaningful representations and foster generalization has caught many concerns. However, the semantical information in language instructions is usually entangled with task-specific state information, which hampers the learning of semantically invariant and reusable representations. In this paper, we propose a method to learn such representations called element randomization, which extracts task-relevant but environment-agnostic semantics from instructions using a set of environments with randomized elements, e.g., topological structures or textures, yet the same language instruction. We theoretically prove the feasibility of learning semantically invariant representations through randomization. In practice, we accordingly develop a hierarchy of policies, where a high-level policy is designed to modulate the behavior of a goal-conditioned low-level policy by proposing subgoals as semantically invariant representations. Experiments on challenging long-horizon tasks show that (1) our low-level policy reliably generalizes to tasks against environment changes; (2) our hierarchical policy exhibits extensible generalization in unseen new tasks that can be decomposed into several solvable sub-tasks; and (3) by storing and replaying language trajectories as succinct policy representations, the agent can complete tasks in a one-shot fashion, i.e., once one successful trajectory has been attained.