# GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation

arXiv: https://arxiv.org/abs/2512.16853 · HTML: https://mlsanity.com/paper/2512.16853/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 39 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation](https://mlsanity.com/paper/2404.14396/) (2024)
- [Transfer between Modalities with MetaQueries](https://mlsanity.com/paper/2504.06256/) (2025)
- [WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2503.07265/) (2025)
- [Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2310.18235/) (2023)
- [GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation](https://mlsanity.com/paper/2406.13743/) (2024)
- [DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation](https://mlsanity.com/paper/2406.16855/) (2024)

## Built on by (most cited first, 30 of 52)

- [GEMS: Agent-Native Multimodal Generation with Memory and Skills](https://mlsanity.com/paper/2603.28088/) (2026)
- [LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model](https://mlsanity.com/paper/2604.02097/) (2026)
- [Context Unrolling in Omni Models](https://mlsanity.com/paper/2604.21921/) (2026)
- [Reinforcing Few-step Generators via Reward-Tilted Distribution Matching](https://mlsanity.com/paper/2605.26108/) (2026)
- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models](https://mlsanity.com/paper/2606.11025/) (2026)
- [COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows](https://mlsanity.com/paper/2607.01709/) (2026)
- [GenRouter: Unified Workflow Routing for Agentic Image Generation](https://mlsanity.com/paper/2608.16721/) (2026)
- [MultiBind: A Benchmark for Attribute Misbinding in Multi-Subject Generation](https://mlsanity.com/paper/2603.21937/) (2026)
- [Steering Visual Generation in Unified Multimodal Models with Understanding Supervision](https://mlsanity.com/paper/2605.05781/) (2026)
- [EPIC: Efficient Predicate-Guided Inference-Time Control for Compositional Text-to-Image Generation](https://mlsanity.com/paper/2605.11722/) (2026)
- [Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimization](https://mlsanity.com/paper/2605.28615/) (2026)
- [PaintBench: Deterministic Evaluation of Precise Visual Editing](https://mlsanity.com/paper/2606.00188/) (2026)
- [i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models](https://mlsanity.com/paper/2606.11289/) (2026)
- [RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space](https://mlsanity.com/paper/2606.14700/) (2026)
- [SymbOmni: Evolving Agentic Omni Models via Symbolic Concept Learning](https://mlsanity.com/paper/2607.12042/) (2026)
- [Scaling Properties of Text Conditioning in Visual Generation](https://mlsanity.com/paper/2607.29679/) (2026)
- [Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System](https://mlsanity.com/paper/2609.01607/) (2026)
- [OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning](https://mlsanity.com/paper/2609.16057/) (2026)
- [Correlation-Weighted Multi-Reward Optimization for Compositional Generation](https://mlsanity.com/paper/2603.18528/) (2026)
- [LENS: Low-Frequency Eigen Noise Shaping for Efficient Diffusion Sampling](https://mlsanity.com/paper/2605.07253/) (2026)
- [FAGER: Factually Grounded Evaluation and Refinement of Text-to-Image Models](https://mlsanity.com/paper/2605.19111/) (2026)
- [DynEval: Holistic Evaluations of T2I Generative Models in the Wild](https://mlsanity.com/paper/2607.11199/) (2026)
- [Inference-Time Scaling of Diffusion Models via Progressive Seed Pruning](https://mlsanity.com/paper/2607.21591/) (2026)
- [Scaling Muon for Diffusion Transformers](https://mlsanity.com/paper/2608.20818/) (2026)
- [Calibrate What You SHIP: Post-Selection Risk Control for Verifier-Guided Text-to-Image Generation](https://mlsanity.com/paper/2608.21748/) (2026)
- [Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Prompts](https://mlsanity.com/paper/2609.35641/) (2026)
- [Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision](https://mlsanity.com/paper/2609.35718/) (2026)
- [DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models](https://mlsanity.com/paper/2605.06170/) (2026)
- [Flow-Direct: Feedback-Efficient and Reusable Guidance for Flow Models via Non-Parametric Guidance Field](https://mlsanity.com/paper/2605.16348/) (2026)

## Abstract (arXiv)

Automating Text-to-Image (T2I) model evaluation is challenging; a judge model must be used to score correctness, and test prompts must be selected to be challenging for current T2I models but not the judge. We argue that satisfying these constraints can lead to benchmark drift over time, where the static benchmark judges fail to keep up with newer model capabilities. We show that benchmark drift is a significant problem for GenEval, one of the most popular T2I benchmarks. Although GenEval was well-aligned with human judgment at the time of its release, it has drifted far from human judgment over time -- resulting in an absolute error of as much as 17.7% for current models. This level of drift strongly suggests that GenEval has been saturated for some time, as we verify via a large-scale human study. To help fill this benchmarking gap, we introduce a new benchmark, GenEval 2, with improved coverage of primitive visual concepts and higher degrees of compositionality, which we show is more challenging for current models. We also introduce Soft-TIFA, an evaluation method for GenEval 2 that combines judgments for visual primitives, which we show is more well-aligned with human judgment and argue is less likely to drift from human-alignment over time (as compared to more holistic judges such as VQAScore). Although we hope GenEval 2 will provide a strong benchmark for many years, avoiding benchmark drift is far from guaranteed and our work, more generally, highlights the importance of continual audits and improvement for T2I and related automated model evaluation benchmarks.