mlsanityData to 7 Oct 2026

Things not Written in Text: Exploring Spatial Commonsense from Visual Signals

arXiv 2203.08075 ↗First posted 15 Mar 2022cs.CL cs.CVcited by 41

Not on the map. Why?

Abstract (arXiv)

Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part of commonsense knowledge. Although pretrained language models (PLMs) succeed in many NLP tasks, they are shown to be ineffective in spatial commonsense reasoning. Starting from the observation that images are more likely to exhibit spatial commonsense than texts, we explore whether models with visual signals learn more spatial commonsense than text-based PLMs. We propose a spatial commonsense benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects under different actions. We probe PLMs and models with visual signals, including vision-language pretrained models and image synthesis models, on this benchmark, and find that image synthesis models are more capable of learning accurate and consistent spatial knowledge than other models. The spatial knowledge from image synthesis models also helps in natural language understanding tasks that require spatial commonsense.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

32 of 48 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
20 more
  1. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  3. 2019How Can We Know What Language Models Know?cited by 953
  4. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  5. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  6. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  7. 2019Abductive Commonsense Reasoningcited by 300
  8. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  9. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  10. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  11. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  12. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  13. 2020Do Language Embeddings Capture Scales?cited by 63
  14. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  15. 2017Verb Physics: Relative Physical Knowledge of Actions and Objectscited by 26
  16. 2019How Large Are Lions? Inducing Distributions over Quantitative Attributescited by 26
  17. 2016Are Elephants Bigger than Butterflies? Reasoning about Sizes of Objectscited by 22
  18. 2017Acquiring Common Sense Spatial Knowledge through Implicit Spatial Templatescited by 10
  19. 2020Beyond Language: Learning Commonsense from Images for Reasoningcited by 3
  20. 2021Visual Probing: Cognitive Framework for Explaining Self-Supervised Image Representationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 12; 2024: 11; 2025: 11; 2026: 1.

41 papers.

Sort:

By area:

  1. 2022Visual Spatial Reasoningcited by 288
  2. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  4. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  5. 2023Multimodal Procedural Planning via Dual Text-Image Promptingcited by 40
  6. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  7. 2023Dissociating language and thought in large language modelscited by 27
  8. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  9. 2023MPCHAT: Towards Multimodal Persona-Grounded Conversationcited by 24
  10. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  11. 2026Large Language Model Reasoning Failurescited by 20
  12. 2022Imagination-Augmented Natural Language Understandingcited by 19
29 more
  1. 2023Can Language Models Understand Physical Concepts?cited by 19
  2. 2022Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imaginationcited by 18
  3. 2024GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoningcited by 7
  4. 2024Will GPT-4 Run DOOM?cited by 6
  5. 2022Visually-augmented pretrained language models for NLP tasks without imagescited by 5
  6. 2023Substance or Style: What Does Your Image Embedding Know?cited by 5
  7. 2024VCD: A Dataset for Visual Commonsense Discovery in Imagescited by 5
  8. 2022VIPHY: Probing "Visible" Physical Commonsense Knowledgecited by 4
  9. 2023ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categoriescited by 4
  10. 2023Probing Physical Reasoning with Counter-Commonsense Contextcited by 4
  11. 2023Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models for Learning Conceptual Spacescited by 3
  12. 2024TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Textcited by 3
  13. 2025MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agentscited by 3
  14. 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarioscited by 3
  15. 2024Ranking Entities along Conceptual Space Dimensions with LLMs: An Analysis of Fine-Tuning Strategiescited by 2
  16. 2024AudioBERT: Audio Knowledge Augmented Language Modelcited by 2
  17. 2025Imagine to Hear: Auditory Knowledge Generation can be an Effective Assistant for Language Modelscited by 2
  18. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2
  19. 2025Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Modelscited by 2
  20. 2023Movie Box Office Prediction With Self-Supervised and Visually Grounded Pretrainingcited by 1
  21. 2024A Note on Bias to Completecited by 1
  22. 2024WinoViz: Probing Visual Properties of Objects Under Different Statescited by 1
  23. 2024Grounding Spatial Relations in Text-Only Language Modelscited by 1
  24. 2024Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Imagecited by 1
  25. 2025AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?cited by 1
  26. 2024Improving the Efficiency of Visually Augmented Language Modelscited by 0
  27. 2025Copycat vs. Original: Multi-modal Pretraining and Variable Importance in Box-office Predictioncited by 0
  28. 2025Extracting Conceptual Spaces from LLMs Using Prototype Embeddingscited by 0
  29. 2025Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning
  2. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  3. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  4. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
  5. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning
  6. 2023Improving Vision-and-Language Reasoning via Spatial Relations Modeling
  7. 2026SpatialText: A Pure-Text Cognitive Benchmark for Spatial Understanding in Large Language Models
  8. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities