# Compositional Learning of Visually-Grounded Concepts Using Reinforcement

arXiv: https://arxiv.org/abs/2309.04504 · HTML: https://mlsanity.com/paper/2309.04504/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 60 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)
- [Task-Driven Modular Networks for Zero-Shot Compositional Learning](https://mlsanity.com/paper/1905.05908/) (2019)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [Detecting Human-Object Interaction via Fabricated Compositional Learning](https://mlsanity.com/paper/2103.08214/) (2021)
- [Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner](https://mlsanity.com/paper/1607.08723/) (2016)
- [Compositional Learning of Image-Text Query for Image Retrieval](https://mlsanity.com/paper/2006.11149/) (2020)
- [Grounded Language Learning Fast and Slow](https://mlsanity.com/paper/2009.01719/) (2020)
- [Abstract Visual Reasoning with Tangram Shapes](https://mlsanity.com/paper/2211.16492/) (2022)
- [Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others](https://mlsanity.com/paper/2102.11938/) (2021)
- [From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence](https://mlsanity.com/paper/2110.15245/) (2021)
- [Teaching Compositionality to CNNs](https://mlsanity.com/paper/1706.04313/) (2017)
- [Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning](https://mlsanity.com/paper/2110.02370/) (2021)
- [Zero-Shot Compositional Concept Learning](https://mlsanity.com/paper/2107.05176/) (2021)
- [Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks](https://mlsanity.com/paper/1901.10113/) (2019)

## Built on by (most cited first, 2 of 2)

- [TangramSR: Can Vision-Language Models Reason in Continuous Geometric Space?](https://mlsanity.com/paper/2602.05570/) (2026)
- [Human-like compositional learning of visually-grounded concepts using synthetic environments](https://mlsanity.com/paper/2504.06618/) (2025)

## Abstract (arXiv)

Children can rapidly generalize compositionally-constructed rules to unseen test sets. On the other hand, deep reinforcement learning (RL) agents need to be trained over millions of episodes, and their ability to generalize to unseen combinations remains unclear. Hence, we investigate the compositional abilities of RL agents, using the task of navigating to specified color-shape targets in synthetic 3D environments. First, we show that when RL agents are naively trained to navigate to target color-shape combinations, they implicitly learn to decompose the combinations, allowing them to (re-)compose these and succeed at held-out test combinations ("compositional learning"). Second, when agents are pretrained to learn invariant shape and color concepts ("concept learning"), the number of episodes subsequently needed for compositional learning decreased by 20 times. Furthermore, only agents trained on both concept and compositional learning could solve a more complex, out-of-distribution environment in zero-shot fashion. Finally, we verified that only text encoders pretrained on image-text datasets (e.g. CLIP) reduced the number of training episodes needed for our agents to demonstrate compositional learning, and also generalized to 5 unseen colors in zero-shot fashion. Overall, our results are the first to demonstrate that RL agents can be trained to implicitly learn concepts and compositionality, to solve more complex environments in zero-shot fashion.