mlsanityData to 7 Oct 2026

Benchmarking Spatial Relationships in Text-to-Image Generation

arXiv 2212.10015 ↗First posted 20 Dec 2022cs.CV cs.AI cs.CLcited by 96

Not on the map. Why?

Cited by 33 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Spatial understanding is a fundamental aspect of computer vision and integral for human-level reasoning about images, making it an important component for grounded language understanding. While recent text-to-image synthesis (T2I) models have shown unprecedented improvements in photorealism, it is unclear whether they have reliable spatial understanding capabilities. We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image. To benchmark existing models, we introduce a dataset, SR2D, that contains sentences describing two or more objects and the spatial relationships between them. We construct an automated evaluation pipeline to recognize objects and their spatial relationships, and employ it in a large-scale evaluation of T2I models. Our experiments reveal a surprising finding that, although state-of-the-art T2I models exhibit high image quality, they are severely limited in their ability to generate multiple objects or the specified spatial relations between them. Our analyses demonstrate several biases and artifacts of T2I models such as the difficulty with generating multiple objects, a bias towards generating the first object mentioned, spatially inconsistent outputs for equivalent relationships, and a correlation between object co-occurrence and spatial understanding capabilities. We conduct a human study that shows the alignment between VISOR and human judgement about spatial understanding. We offer the SR2D dataset and the VISOR metric to the community in support of T2I reasoning research.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

39 of 56 references.

References · 39

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2016Improved Techniques for Training GANscited by 2,938
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
27 more
  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  2. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  3. 2016Generative Adversarial Text to Image Synthesiscited by 735
  4. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  5. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  6. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  7. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  8. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  9. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  10. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  11. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  12. 2022Visual Spatial Reasoningcited by 288
  13. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  14. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  15. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  16. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  17. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  18. 2022A very preliminary analysis of DALL-E 2cited by 82
  19. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  20. 2020Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?cited by 69
  21. 2022Testing Relational Understanding in Text-Guided Image Generationcited by 59
  22. 2020VQA-LOL: Visual Question Answering under the Lens of Logiccited by 48
  23. 2019Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generationcited by 36
  24. 2022DALL-E 2 Fails to Reliably Capture Common Syntactic Processescited by 31
  25. 2018Learning Hierarchical Semantic Image Manipulation through Structured Representationscited by 27
  26. 2020Learning Object Placements For Relational Instructions by Hallucinating Scene Representationscited by 16
  27. 2020IQ-VQA: Intelligent Visual Question Answeringcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 28; 2024: 37; 2025: 20; 2026: 11.

96 papers.

Sort:

By area:

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  3. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  4. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  5. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  6. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  7. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  8. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  9. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  10. 2023Controllable Text-to-Image Generation with GPT-4cited by 58
  11. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
  12. 2023EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methodscited by 52
38 more
  1. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  2. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  3. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  4. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  5. 2023Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Modelscited by 36
  6. 2023Text encoders bottleneck compositionality in contrastive vision-language modelscited by 34
  7. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  8. 2024Getting it Right: Improving Spatial Consistency in Text-to-Image Modelscited by 30
  9. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generationcited by 29
  10. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  11. 2023Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Searchcited by 25
  12. 2023Auditing Gender Presentation Differences in Text-to-Image Modelscited by 22
  13. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 21
  14. 2023Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generationcited by 19
  15. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learningcited by 19
  16. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 19
  17. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Datacited by 19
  18. 2023TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generationcited by 18
  19. 2023Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignmentcited by 18
  20. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 17
  21. 2024Spatial-Aware Latent Initialization for Controllable Image Generationcited by 15
  22. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  23. 2025Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?cited by 15
  24. 2023From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Spurious Correlations in Image Recognitioncited by 12
  25. 2023A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metricscited by 12
  26. 2024LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotationscited by 12
  27. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11
  28. 2025Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulationcited by 11
  29. 2023ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Modelscited by 10
  30. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Modelscited by 10
  31. 2023LucidDreaming: Controllable Object-Centric 3D Generationcited by 10
  32. 2025Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesiscited by 10
  33. 2024Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Modelscited by 9
  34. 2024Text-to-Image Synthesis: A Decade Surveycited by 9
  35. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLMcited by 9
  36. 2024Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generationcited by 8
  37. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  38. 2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 8

50 most cited of 96 All 96 (JSON)