# Evaluating the Generation of Spatial Relations in Text and Image Generative Models

arXiv: https://arxiv.org/abs/2411.07664 · HTML: https://mlsanity.com/paper/2411.07664/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [Generating Images with Multimodal Language Models](https://mlsanity.com/paper/2305.17216/) (2023)
- [T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation](https://mlsanity.com/paper/2307.06350/) (2023)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.13655/) (2023)
- [DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models](https://mlsanity.com/paper/2202.04053/) (2022)
- [Holistic Evaluation of Text-To-Image Models](https://mlsanity.com/paper/2311.04287/) (2023)
- [Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis](https://mlsanity.com/paper/1801.05091/) (2018)
- [Compositional 3D Scene Generation using Locally Conditioned Diffusion](https://mlsanity.com/paper/2303.12218/) (2023)
- [Benchmarking Spatial Relationships in Text-to-Image Generation](https://mlsanity.com/paper/2212.10015/) (2022)
- [LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation](https://mlsanity.com/paper/2305.11116/) (2023)
- [Evaluating Spatial Understanding of Large Language Models](https://mlsanity.com/paper/2310.14540/) (2023)
- [Testing Relational Understanding in Text-Guided Image Generation](https://mlsanity.com/paper/2208.00005/) (2022)
- [The Preposition Project](https://mlsanity.com/paper/2104.08922/) (2021)

## Built on by (most cited first, 6 of 6)

- [WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2503.07265/) (2025)
- [ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memory](https://mlsanity.com/paper/2510.06664/) (2025)
- [GenSpace: Benchmarking Spatially-Aware Image Generation](https://mlsanity.com/paper/2505.24870/) (2025)
- [Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation ability](https://mlsanity.com/paper/2504.08003/) (2025)
- [Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Image](https://mlsanity.com/paper/2509.02295/) (2025)
- [Stitch: Training-Free Position Control in Multimodal Diffusion Transformers](https://mlsanity.com/paper/2509.26644/) (2025)

## Abstract (arXiv)

Understanding spatial relations is a crucial cognitive ability for both humans and AI. While current research has predominantly focused on the benchmarking of text-to-image (T2I) models, we propose a more comprehensive evaluation that includes textitboth T2I and Large Language Models (LLMs). As spatial relations are naturally understood in a visuo-spatial manner, we develop an approach to convert LLM outputs into an image, thereby allowing us to evaluate both T2I models and LLMs textitvisually. We examined the spatial relation understanding of 8 prominent generative models (3 T2I models and 5 LLMs) on a set of 10 common prepositions, as well as assess the feasibility of automatic evaluation methods. Surprisingly, we found that T2I models only achieve subpar performance despite their impressive general image-generation abilities. Even more surprisingly, our results show that LLMs are significantly more accurate than T2I models in generating spatial relations, despite being primarily trained on textual data. We examined reasons for model failures and highlight gaps that can be filled to enable more spatially faithful generations.