# GIR-Bench: Versatile Benchmark for Generating Images with Reasoning

arXiv: https://arxiv.org/abs/2510.11026 · HTML: https://mlsanity.com/paper/2510.11026/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [OmniGen2: Towards Instruction-Aligned Multimodal Generation](https://mlsanity.com/paper/2506.18871/) (2025)
- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)
- [Transfer between Modalities with MetaQueries](https://mlsanity.com/paper/2504.06256/) (2025)
- [ImgEdit: A Unified Image Editing Dataset and Benchmark](https://mlsanity.com/paper/2505.20275/) (2025)
- [WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2503.07265/) (2025)
- [In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer](https://mlsanity.com/paper/2504.20690/) (2025)
- [PaddleOCR 3.0 Technical Report](https://mlsanity.com/paper/2507.05595/) (2025)
- [HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer](https://mlsanity.com/paper/2505.22705/) (2025)
- [Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing](https://mlsanity.com/paper/2504.02826/) (2025)
- [Ovis-U1 Technical Report](https://mlsanity.com/paper/2506.23044/) (2025)
- [UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation](https://mlsanity.com/paper/2506.03147/) (2025)
- [Liquid: Language Models are Scalable and Unified Multi-modal Generators](https://mlsanity.com/paper/2412.04332/) (2024)
- [Reconstruction Alignment Improves Unified Multimodal Models](https://mlsanity.com/paper/2509.07295/) (2025)
- [VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model](https://mlsanity.com/paper/2501.12327/) (2025)
- [Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model](https://mlsanity.com/paper/2509.04548/) (2025)
- [T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation](https://mlsanity.com/paper/2508.17472/) (2025)
- [Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?](https://mlsanity.com/paper/2509.03516/) (2025)

## Built on by (most cited first, 15 of 15)

- [MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation](https://mlsanity.com/paper/2512.14691/) (2025)
- [Quantifying the Gap between Understanding and Generation within Unified Multimodal Models](https://mlsanity.com/paper/2602.02140/) (2026)
- [Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward](https://mlsanity.com/paper/2511.20561/) (2025)
- [MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models](https://mlsanity.com/paper/2602.19497/) (2026)
- [MetaPoint: Unlocking Precise Spatial Control in Agentic Visual Generation](https://mlsanity.com/paper/2606.05031/) (2026)
- [CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions](https://mlsanity.com/paper/2603.26174/) (2026)
- [Gloria: Consistent Character Video Generation via Content Anchors](https://mlsanity.com/paper/2603.29931/) (2026)
- [AEGIS: Exploring the Limit of World Knowledge Capabilities for Unified Mulitmodal Models](https://mlsanity.com/paper/2601.00561/) (2026)
- [End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction](https://mlsanity.com/paper/2603.14435/) (2026)
- [SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models](https://mlsanity.com/paper/2606.23041/) (2026)
- [Image-Space Rule Discovery](https://mlsanity.com/paper/2608.00490/) (2026)
- [Thinking in Pictures: A Systematic Benchmark for Reasoning-driven Image Generation](https://mlsanity.com/paper/2609.02864/) (2026)
- [Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency](https://mlsanity.com/paper/2606.10620/) (2026)
- [Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning](https://mlsanity.com/paper/2606.24548/) (2026)
- [A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems](https://mlsanity.com/paper/2608.11907/) (2026)

## Abstract (arXiv)

Unified multimodal models integrate the reasoning capacity of large language models with both image understanding and generation, showing great promise for advanced multimodal intelligence. However, the community still lacks a rigorous reasoning-centric benchmark to systematically evaluate the alignment between understanding and generation, and their generalization potential in complex visual tasks. To this end, we introduce GIR-Bench, a comprehensive benchmark that evaluates unified models across three complementary perspectives. Firstly, we investigate understanding-generation consistency (GIR-Bench-UGC), asking whether models can consistently leverage the same knowledge in both understanding and generation tasks. Secondly, we investigate whether models can perform reasoning-centric text-to-image generation that requires applying logical constraints and implicit knowledge to generate faithful visual content (GIR-Bench-T2I). Thirdly, we evaluate whether models can handle multi-step reasoning in editing (GIR-Bench-Edit). For each subset, we carefully design different task-specific evaluation pipelines tailored for each task. This enables fine-grained and interpretable evaluation while mitigating biases from the prevalent MLLM-as-a-Judge paradigm. Extensive ablations over various unified models and generation-only systems have shown that: Although unified models are more capable of reasoning-driven visual tasks, they still exhibit a persistent gap between understanding and generation. The data and code for GIR-Bench are available at https://github.com/HKUST-LongGroup/GIR-Bench.