mlsanityData to 7 Oct 2026

ShapeWorld - A new test methodology for multimodal language understanding

arXiv 1704.04517 ↗First posted 14 Apr 2017cs.CL cs.AI cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter's specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require true generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating various visual question answering models on four different tasks, and show how our framework gives us detailed insights into their capabilities and limitations. By open-sourcing our framework, we hope to stimulate progress in the field of multimodal language understanding.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 4; 2023: 5; 2024: 11; 2025: 6; 2026: 3.

37 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  4. 2021Emergent Communication of Generalizationscited by 41
  5. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  6. 2024VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Informationcited by 39
  7. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  8. 2024Do Vision-Language Models Really Understand Visual Language?cited by 29
  9. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
  10. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  11. 2021Open-domain clarification question generation without question examplescited by 20
  12. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
25 more
  1. 2021Rich Semantics Improve Few-shot Learningcited by 15
  2. 2022Name Your Style: An Arbitrary Artist-aware Image Style Transfercited by 14
  3. 2022RelViT: Concept-guided Vision Transformer for Visual Relational Reasoningcited by 14
  4. 2023Scalable Neural-Probabilistic Answer Set Programmingcited by 14
  5. 2024Visually Descriptive Language Model for Vector Graphics Reasoningcited by 14
  6. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysiscited by 14
  7. 2021Calibrate your listeners! Robust communication-based training for pragmatic speakerscited by 9
  8. 2021SLASH: Embracing Probabilistic Circuits into Neural Answer Set Programmingcited by 8
  9. 2024Understanding Depth and Height Perception in Large Visual-Language Modelscited by 8
  10. 2023Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lackingcited by 7
  11. 2024Concept-Best-Matching: Evaluating Compositionality in Emergent Communicationcited by 7
  12. 2025Chimera: Diagnosing Shortcut Learning in Visual-Language Understandingcited by 6
  13. 2021KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligencecited by 5
  14. 2021TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoningcited by 4
  15. 2022Explanatory Learning: Beyond Empiricism in Neural Networkscited by 3
  16. 2024Communicating with Speakers and Listeners of Different Pragmatic Levelscited by 2
  17. 2024Semantics and Spatiality of Emergent Communicationcited by 2
  18. 2024XferBench: a Data-Driven Benchmark for Emergent Languagecited by 1
  19. 2026CtD: Composition through Decomposition in Emergent Communicationcited by 1
  20. 2022Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoningcited by 0
  21. 2024Bridging Text and Image for Artist Style Transfer via Contrastive Learningcited by 0
  22. 2025VQEL: Enabling Self-Play in Emergent Language Games via Agent-Internal Vector Quantizationcited by 0
  23. 2025Morpheme Induction for Emergent Languagecited by 0
  24. 2026Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Imagescited by 0
  25. 2026When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenescited by 0