# Evaluating Numerical Reasoning in Text-to-Image Models

arXiv: https://arxiv.org/abs/2406.14774 · HTML: https://mlsanity.com/paper/2406.14774/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation](https://mlsanity.com/paper/2307.06350/) (2023)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2310.18235/) (2023)
- [AnyText: Multilingual Visual Text Generation And Editing](https://mlsanity.com/paper/2311.03054/) (2023)
- [Holistic Evaluation of Text-To-Image Models](https://mlsanity.com/paper/2311.04287/) (2023)
- [TallyQA: Answering Complex Counting Questions](https://mlsanity.com/paper/1810.12440/) (2018)
- [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://mlsanity.com/paper/2402.14903/) (2024)
- [What You See is What You Read? Improving Text-Image Alignment Evaluation](https://mlsanity.com/paper/2305.10400/) (2023)
- [Visual Programming for Text-to-Image Generation and Evaluation](https://mlsanity.com/paper/2305.15328/) (2023)
- [Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings](https://mlsanity.com/paper/2404.16820/) (2024)
- [DALL-E 2 Fails to Reliably Capture Common Syntactic Processes](https://mlsanity.com/paper/2210.12889/) (2022)
- [Interpretable Counting for Visual Question Answering](https://mlsanity.com/paper/1712.08697/) (2017)
- [Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models](https://mlsanity.com/paper/2405.16759/) (2024)

## Built on by (most cited first, 12 of 12)

- [Dynamic Classifier-Free Diffusion Guidance via Online Feedback](https://mlsanity.com/paper/2509.16131/) (2025)
- [GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensics](https://mlsanity.com/paper/2603.11442/) (2026)
- [Sequential Enumeration in Large Language Models](https://mlsanity.com/paper/2512.04727/) (2025)
- [Causality neq Decodability, and Vice Versa: Lessons from Interpreting Counting ViTs](https://mlsanity.com/paper/2510.09794/) (2025)
- [Demystifying Numerosity in Diffusion Models -- Limitations and Remedies](https://mlsanity.com/paper/2510.11117/) (2025)
- [FireScope: Wildfire Risk Raster Prediction with a Chain-of-Thought Oracle](https://mlsanity.com/paper/2511.17171/) (2025)
- [Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generation](https://mlsanity.com/paper/2505.18730/) (2025)
- [The MixCount Dataset: Bridging the Data Gap for Open-Vocabulary Object Counting](https://mlsanity.com/paper/2605.18063/) (2026)
- [Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Prompts](https://mlsanity.com/paper/2609.35641/) (2026)
- [Instruct-ICL: Instruction-Guided In-Context Learning for Post-Disaster Damage Assessment](https://mlsanity.com/paper/2605.11439/) (2026)
- [Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models](https://mlsanity.com/paper/2606.05949/) (2026)
- [Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning](https://mlsanity.com/paper/2606.24548/) (2026)

## Abstract (arXiv)

Text-to-image generative models are capable of producing high-quality images that often faithfully depict concepts described using natural language. In this work, we comprehensively evaluate a range of text-to-image models on numerical reasoning tasks of varying difficulty, and show that even the most advanced models have only rudimentary numerical skills. Specifically, their ability to correctly generate an exact number of objects in an image is limited to small numbers, it is highly dependent on the context the number term appears in, and it deteriorates quickly with each successive number. We also demonstrate that models have poor understanding of linguistic quantifiers (such as "a few" or "as many as"), the concept of zero, and struggle with more advanced concepts such as partial quantities and fractional representations. We bundle prompts, generated images and human annotations into GeckoNum, a novel benchmark for evaluation of numerical reasoning.