# ShapeWorld - A new test methodology for multimodal language understanding

arXiv: https://arxiv.org/abs/1704.04517 · HTML: https://mlsanity.com/paper/1704.04517/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2017.

## Built on by (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Emergent Communication of Generalizations](https://mlsanity.com/paper/2106.02668/) (2021)
- [VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information](https://mlsanity.com/paper/2412.00947/) (2024)
- [Do Vision-Language Models Really Understand Visual Language?](https://mlsanity.com/paper/2410.00193/) (2024)
- [What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation](https://mlsanity.com/paper/2406.16320/) (2024)
- [iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs](https://mlsanity.com/paper/2502.03214/) (2025)
- [Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation](https://mlsanity.com/paper/2506.21876/) (2025)
- [Open-domain clarification question generation without question examples](https://mlsanity.com/paper/2110.09779/) (2021)
- [MEWL: Few-shot multimodal word learning with referential uncertainty](https://mlsanity.com/paper/2306.00503/) (2023)
- [Rich Semantics Improve Few-shot Learning](https://mlsanity.com/paper/2104.12709/) (2021)
- [RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning](https://mlsanity.com/paper/2204.11167/) (2022)
- [Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis](https://mlsanity.com/paper/2409.00106/) (2024)
- [Name Your Style: An Arbitrary Artist-aware Image Style Transfer](https://mlsanity.com/paper/2202.13562/) (2022)
- [Scalable Neural-Probabilistic Answer Set Programming](https://mlsanity.com/paper/2306.08397/) (2023)
- [Visually Descriptive Language Model for Vector Graphics Reasoning](https://mlsanity.com/paper/2404.06479/) (2024)
- [Calibrate your listeners! Robust communication-based training for pragmatic speakers](https://mlsanity.com/paper/2110.05422/) (2021)
- [SLASH: Embracing Probabilistic Circuits into Neural Answer Set Programming](https://mlsanity.com/paper/2110.03395/) (2021)
- [Understanding Depth and Height Perception in Large Visual-Language Models](https://mlsanity.com/paper/2408.11748/) (2024)
- [Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking](https://mlsanity.com/paper/2307.16806/) (2023)
- [Concept-Best-Matching: Evaluating Compositionality in Emergent Communication](https://mlsanity.com/paper/2403.14705/) (2024)
- [Chimera: Diagnosing Shortcut Learning in Visual-Language Understanding](https://mlsanity.com/paper/2509.22437/) (2025)
- [KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence](https://mlsanity.com/paper/2103.00519/) (2021)
- [TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning](https://mlsanity.com/paper/2111.10756/) (2021)
- [Explanatory Learning: Beyond Empiricism in Neural Networks](https://mlsanity.com/paper/2201.10222/) (2022)
- [Communicating with Speakers and Listeners of Different Pragmatic Levels](https://mlsanity.com/paper/2410.05851/) (2024)
- [Semantics and Spatiality of Emergent Communication](https://mlsanity.com/paper/2411.10173/) (2024)
- [XferBench: a Data-Driven Benchmark for Emergent Language](https://mlsanity.com/paper/2407.03456/) (2024)

## Abstract (arXiv)

We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter's specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require true generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating various visual question answering models on four different tasks, and show how our framework gives us detailed insights into their capabilities and limitations. By open-sourcing our framework, we hope to stimulate progress in the field of multimodal language understanding.