mlsanityData to 7 Oct 2026

A Benchmark for Systematic Generalization in Grounded Language Understanding

arXiv 2003.05161 ↗First posted 11 Mar 2020cs.CL cs.AI cs.LGcited by 108

Not on the map. Why?

Abstract (arXiv)

Humans easily interpret expressions that describe unfamiliar situations composed from familiar parts ("greet the pink brontosaurus by the ferris wheel"). Modern neural networks, by contrast, struggle to interpret novel compositions. In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding. Going beyond a related benchmark that focused on syntactic aspects of generalization, gSCAN defines a language grounded in the states of a grid world, facilitating novel evaluations of acquiring linguistically motivated rules. For example, agents must understand how adjectives such as 'small' are interpreted relative to the current world state or how adverbs such as 'cautiously' combine with new verbs. We test a strong multi-modal baseline model and a state-of-the-art compositional method finding that, in most cases, they fail dramatically when generalization requires systematic compositional rules.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 25; 2023: 18; 2024: 15; 2025: 9; 2026: 14.

108 papers.

Sort:

By area:

  1. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  2. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  3. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  4. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  5. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 69
  6. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  7. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  8. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  9. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoningcited by 49
  10. 2021PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D Worldcited by 47
  11. 2024LLM-A*: Large Language Model Enhanced Incremental Heuristic Search on Path Planningcited by 44
  12. 2022A Benchmark for Compositional Visual Reasoningcited by 43
38 more
  1. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  2. 2021ACRE: Abstract Causal REasoning Beyond Covariationcited by 30
  3. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  4. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  5. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  6. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  7. 2023ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretationcited by 24
  8. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  9. 2021ReaSCAN: Compositional Reasoning in Language Groundingcited by 19
  10. 2021Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Taskscited by 19
  11. 2025A Survey on Large Language Models for Automated Planningcited by 17
  12. 2021How Modular Should Neural Module Networks Be for Systematic Generalization?cited by 15
  13. 2021Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?cited by 15
  14. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  15. 2022RelViT: Concept-guided Vision Transformer for Visual Relational Reasoningcited by 14
  16. 2024On The Specialization of Neural Modulescited by 14
  17. 2021Quantifying Adaptability in Pre-trained Language Models with 500 Taskscited by 13
  18. 2022Improving Systematic Generalization Through Modularity and Augmentationcited by 13
  19. 2022Benchmarking Compositionality with Formal Languagescited by 12
  20. 2022Neural-Symbolic Recursive Machine for Systematic Generalizationcited by 12
  21. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  22. 2022When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarkscited by 11
  23. 2021SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmarkcited by 10
  24. 2024Baba Is AI: Break the Rules to Beat the Benchmarkcited by 10
  25. 2021Compositional Generalization in Multilingual Semantic Parsing over Wikidatacited by 9
  26. 2024Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generationcited by 8
  27. 2025SAGE-Eval: Evaluating LLMs for Systematic Generalizations of Safety Factscited by 8
  28. 2021Grounding Spatio-Temporal Language with Transformerscited by 7
  29. 2021SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agentscited by 7
  30. 2021Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarkingcited by 7
  31. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  32. 2024Look Further Ahead: Testing the Limits of GPT-4 in Path Planningcited by 7
  33. 2021A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semanticscited by 6
  34. 2021Generalization in Multimodal Language Learning from Simulationcited by 6
  35. 2022One-Shot Learning from a Demonstration with Hierarchical Latent Languagecited by 6
  36. 2023Does Deep Learning Learn to Abstract? A Systematic Probing Frameworkcited by 6
  37. 2023Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Modelscited by 6
  38. 2021Robust Generalization of Quadratic Neural Networks via Function Identificationcited by 5

50 most cited of 108 All 108 (JSON)