# Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?

arXiv: https://arxiv.org/abs/2109.12243 · HTML: https://mlsanity.com/paper/2109.12243/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 25 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017)
- [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017)
- [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020)
- [Grounded Language Learning in a Simulated 3D World](https://mlsanity.com/paper/1706.06551/) (2017)
- [Gated-Attention Architectures for Task-Oriented Language Grounding](https://mlsanity.com/paper/1706.07230/) (2017)
- [CLOSURE: Assessing Systematic Generalization of CLEVR Models](https://mlsanity.com/paper/1912.05783/) (2019)
- [Interactive Grounded Language Acquisition and Generalization in a 2D World](https://mlsanity.com/paper/1802.01433/) (2018)
- [Compositional Networks Enable Systematic Generalization for Grounded Language Understanding](https://mlsanity.com/paper/2008.02742/) (2020)
- [Systematic Generalization on gSCAN with Language Conditioned Embedding](https://mlsanity.com/paper/2009.05552/) (2020)
- [Think before you act: A simple baseline for compositional generalization](https://mlsanity.com/paper/2009.13962/) (2020)
- [Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts](https://mlsanity.com/paper/1908.10285/) (2019)

## Built on by (most cited first, 15 of 15)

- [Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text](https://mlsanity.com/paper/2307.07696/) (2023)
- [A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices](https://mlsanity.com/paper/2406.08787/) (2024)
- [Improving Systematic Generalization Through Modularity and Augmentation](https://mlsanity.com/paper/2202.10745/) (2022)
- [When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks](https://mlsanity.com/paper/2210.12786/) (2022)
- [NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language](https://mlsanity.com/paper/2509.25757/) (2025)
- [Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking](https://mlsanity.com/paper/2112.00086/) (2021)
- [Can LLM find the green circle? Investigation and Human-guided tool manipulation for compositional generalization](https://mlsanity.com/paper/2312.07763/) (2023)
- [Look Further Ahead: Testing the Limits of GPT-4 in Path Planning](https://mlsanity.com/paper/2406.12000/) (2024)
- [One-Shot Learning from a Demonstration with Hierarchical Latent Language](https://mlsanity.com/paper/2203.04806/) (2022)
- [Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models](https://mlsanity.com/paper/2311.09064/) (2023)
- [Compositional Generalization in Grounded Language Learning via Induced Model Sparsity](https://mlsanity.com/paper/2207.02518/) (2022)
- [Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments](https://mlsanity.com/paper/2311.04364/) (2023)
- [Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models](https://mlsanity.com/paper/2511.06146/) (2025)
- [A Short Survey of Systematic Generalization](https://mlsanity.com/paper/2211.11956/) (2022)
- [Data Distributional Properties As Inductive Bias for Systematic Generalization](https://mlsanity.com/paper/2502.20499/) (2025)

## Abstract (arXiv)

We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding. First, we study which aspects of the original benchmark can be solved by commonly used methods in multi-modal research. We find that a general-purpose Transformer-based model with cross-modal attention achieves strong performance on a majority of the gSCAN splits, surprisingly outperforming more specialized approaches from prior work. Furthermore, our analysis suggests that many of the remaining errors reveal the same fundamental challenge in systematic generalization of linguistic constructs regardless of visual context. Second, inspired by this finding, we propose challenging new tasks for gSCAN by generating data to incorporate relations between objects in the visual environment. Finally, we find that current models are surprisingly data inefficient given the narrow scope of commands in gSCAN, suggesting another challenge for future work.