# PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models

arXiv: https://arxiv.org/abs/2406.11802 · HTML: https://mlsanity.com/paper/2406.11802/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation](https://mlsanity.com/paper/2403.12015/) (2024)
- [Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs](https://mlsanity.com/paper/2401.11708/) (2024)
- [MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI](https://mlsanity.com/paper/2404.16006/) (2024)
- [Holistic Evaluation of Text-To-Image Models](https://mlsanity.com/paper/2311.04287/) (2023)
- [GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation](https://mlsanity.com/paper/2401.04092/) (2024)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion](https://mlsanity.com/paper/2310.08579/) (2023)
- [Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark](https://mlsanity.com/paper/2211.12112/) (2022)
- [Lighting (In)consistency of Paint by Text](https://mlsanity.com/paper/2207.13744/) (2022)
- [Qualitative Failures of Image Generation Models and Their Application in Detecting Deepfakes](https://mlsanity.com/paper/2304.06470/) (2023)
- [Position: Towards Implicit Prompt For Text-To-Image Models](https://mlsanity.com/paper/2403.02118/) (2024)

## Built on by (most cited first, 30 of 36)

- [WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2503.07265/) (2025)
- [GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation](https://mlsanity.com/paper/2504.02782/) (2025)
- [T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation](https://mlsanity.com/paper/2505.00337/) (2025)
- [T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation](https://mlsanity.com/paper/2508.17472/) (2025)
- [Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?](https://mlsanity.com/paper/2509.03516/) (2025)
- [Rethinking Video Generation Model for the Embodied World](https://mlsanity.com/paper/2601.15282/) (2026)
- [A Survey of Multimodal Hallucination Evaluation and Detection](https://mlsanity.com/paper/2507.19024/) (2025)
- [Mind-Brush: Integrating Agentic Cognitive Search and Reasoning into Image Generation](https://mlsanity.com/paper/2602.01756/) (2026)
- [GenExam: A Multidisciplinary Text-to-Image Exam](https://mlsanity.com/paper/2509.14232/) (2025)
- [T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generation](https://mlsanity.com/paper/2507.18107/) (2025)
- [Impossible Videos](https://mlsanity.com/paper/2503.14378/) (2025)
- [LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference](https://mlsanity.com/paper/2510.11512/) (2025)
- [Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Models](https://mlsanity.com/paper/2601.20354/) (2026)
- [Science-T2I: Addressing Scientific Illusions in Image Synthesis](https://mlsanity.com/paper/2504.13129/) (2025)
- [T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts](https://mlsanity.com/paper/2412.04300/) (2024)
- [Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generation](https://mlsanity.com/paper/2606.26907/) (2026)
- [Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help](https://mlsanity.com/paper/2503.06884/) (2025)
- [MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation](https://mlsanity.com/paper/2512.14691/) (2025)
- [GenSpace: Benchmarking Spatially-Aware Image Generation](https://mlsanity.com/paper/2505.24870/) (2025)
- [Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning](https://mlsanity.com/paper/2507.01908/) (2025)
- [Merge-of-Thought Distillation](https://mlsanity.com/paper/2509.08814/) (2025)
- [Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling](https://mlsanity.com/paper/2604.28185/) (2026)
- [Training-Trajectory-Aware Token Selection](https://mlsanity.com/paper/2601.10348/) (2026)
- [Quality Prediction of AI Generated Images and Videos: Emerging Trends and Opportunities](https://mlsanity.com/paper/2410.08534/) (2024)
- [FireScope: Wildfire Risk Raster Prediction with a Chain-of-Thought Oracle](https://mlsanity.com/paper/2511.17171/) (2025)
- [Envision: Benchmarking Unified Understanding & Generation for Causal World Process Insights](https://mlsanity.com/paper/2512.01816/) (2025)
- [Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generation](https://mlsanity.com/paper/2505.18730/) (2025)
- [Physics-Based Benchmarking Metrics for Multimodal Synthetic Images](https://mlsanity.com/paper/2511.15204/) (2025)
- [EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesis](https://mlsanity.com/paper/2603.20828/) (2026)
- [Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models](https://mlsanity.com/paper/2604.21523/) (2026)

## Abstract (arXiv)

Text-to-image (T2I) models have made substantial progress in generating images from textual prompts. However, they frequently fail to produce images consistent with physical commonsense, a vital capability for applications in world simulation and everyday tasks. Current T2I evaluation benchmarks focus on metrics such as accuracy, bias, and safety, neglecting the evaluation of models' internal knowledge, particularly physical commonsense. To address this issue, we introduce PhyBench, a comprehensive T2I evaluation dataset comprising 700 prompts across 4 primary categories: mechanics, optics, thermodynamics, and material properties, encompassing 31 distinct physical scenarios. We assess 6 prominent T2I models, including proprietary models DALLE3 and Gemini, and demonstrate that incorporating physical principles into prompts enhances the models' ability to generate physically accurate images. Our findings reveal that: (1) even advanced models frequently err in various physical scenarios, except for optics; (2) GPT-4o, with item-specific scoring instructions, effectively evaluates the models' understanding of physical commonsense, closely aligning with human assessments; and (3) current T2I models are primarily focused on text-to-image translation, lacking profound reasoning regarding physical commonsense. We advocate for increased attention to the inherent knowledge within T2I models, beyond their utility as mere image generation tools. The data will be available soon.