# Conceptual Reinforcement Learning for Language-Conditioned Tasks

arXiv: https://arxiv.org/abs/2303.05069 · HTML: https://mlsanity.com/paper/2303.05069/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 38 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 30 of 36)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [Interactive Fiction Games: A Colossal Adventure](https://mlsanity.com/paper/1909.05398/) (2019)
- [DARLA: Improving Zero-Shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/1707.08475/) (2017)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Using Natural Language for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/1903.02020/) (2019)
- [Keep CALM and Explore: Language Models for Action Generation in Text-based Games](https://mlsanity.com/paper/2010.02903/) (2020)
- [Invariant Causal Prediction for Block MDPs](https://mlsanity.com/paper/2003.06016/) (2020)
- [Grounded Language Learning in a Simulated 3D World](https://mlsanity.com/paper/1706.06551/) (2017)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [Decoupling Value and Policy for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2102.10330/) (2021)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Learning to Understand Goal Specifications by Modelling Reward](https://mlsanity.com/paper/1806.01946/) (2018)
- [Gated-Attention Architectures for Task-Oriented Language Grounding](https://mlsanity.com/paper/1706.07230/) (2017)
- [Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics](https://mlsanity.com/paper/1706.04317/) (2017)
- [Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences](https://mlsanity.com/paper/1506.04089/) (2015)
- [Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.07393/) (2021)
- [Learning to Win by Reading Manuals in a Monte-Carlo Framework](https://mlsanity.com/paper/1401.5390/) (2014)
- [How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds](https://mlsanity.com/paper/2006.07409/) (2020)
- [Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines](https://mlsanity.com/paper/2010.03790/) (2020)
- [RTFM: Generalising to Novel Environment Dynamics via Reading](https://mlsanity.com/paper/1910.08210/) (2019)
- [Unsupervised Visual Attention and Invariance for Reinforcement Learning](https://mlsanity.com/paper/2104.02921/) (2021)
- [Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning](https://mlsanity.com/paper/2010.02386/) (2020)
- [Causality-driven Hierarchical Structure Discovery for Reinforcement Learning](https://mlsanity.com/paper/2210.06964/) (2022)
- [Alignment-based compositional semantics for instruction following](https://mlsanity.com/paper/1508.06491/) (2015)
- [Exploration Based Language Learning for Text-Based Games](https://mlsanity.com/paper/2001.08868/) (2020)

## Built on by (most cited first, 8 of 8)

- [LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions](https://mlsanity.com/paper/2405.11106/) (2024)
- [Context Shift Reduction for Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2311.03695/) (2023)
- [Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2311.01075/) (2023)
- [Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignment](https://mlsanity.com/paper/2406.01103/) (2024)
- [Online Prototype Alignment for Few-shot Policy Transfer](https://mlsanity.com/paper/2306.07307/) (2023)
- [Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction](https://mlsanity.com/paper/2402.04154/) (2024)
- [DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2510.19562/) (2025)
- [Conceptual Belief-Informed Reinforcement Learning](https://mlsanity.com/paper/2410.01739/) (2024)

## Abstract (arXiv)

Despite the broad application of deep reinforcement learning (RL), transferring and adapting the policy to unseen but similar environments is still a significant challenge. Recently, the language-conditioned policy is proposed to facilitate policy transfer through learning the joint representation of observation and text that catches the compact and invariant information across environments. Existing studies of language-conditioned RL methods often learn the joint representation as a simple latent layer for the given instances (episode-specific observation and text), which inevitably includes noisy or irrelevant information and cause spurious correlations that are dependent on instances, thus hurting generalization performance and training efficiency. To address this issue, we propose a conceptual reinforcement learning (CRL) framework to learn the concept-like joint representation for language-conditioned policy. The key insight is that concepts are compact and invariant representations in human cognition through extracting similarities from numerous instances in real-world. In CRL, we propose a multi-level attention encoder and two mutual information constraints for learning compact and invariant concepts. Verified in two challenging environments, RTFM and Messenger, CRL significantly improves the training efficiency (up to 70%) and generalization ability (up to 30%) to the new environment dynamics.