# ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing

arXiv: https://arxiv.org/abs/2503.14482 · HTML: https://mlsanity.com/paper/2503.14482/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 59 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)

## Built on by (most cited first, 24 of 24)

- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [Emu3.5: Native Multimodal Models are World Learners](https://mlsanity.com/paper/2510.26583/) (2025)
- [KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models](https://mlsanity.com/paper/2505.16707/) (2025)
- [NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation](https://mlsanity.com/paper/2601.02204/) (2026)
- [GEditBench v2: A Human-Aligned Benchmark for General Image Editing](https://mlsanity.com/paper/2603.28547/) (2026)
- [UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits](https://mlsanity.com/paper/2512.02790/) (2025)
- [Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis](https://mlsanity.com/paper/2602.13028/) (2026)
- [I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models](https://mlsanity.com/paper/2512.04660/) (2025)
- [ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks](https://mlsanity.com/paper/2603.27862/) (2026)
- [TAG-MoE: Task-Aware Gating for Unified Generative Mixture-of-Experts](https://mlsanity.com/paper/2601.08881/) (2026)
- [LaTo: Landmark-tokenized Diffusion Transformer for Fine-grained Human Face Editing](https://mlsanity.com/paper/2509.25731/) (2025)
- [MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models](https://mlsanity.com/paper/2602.19497/) (2026)
- [Are Image-to-Video Models Good Zero-Shot Image Editors?](https://mlsanity.com/paper/2511.19435/) (2025)
- [ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services](https://mlsanity.com/paper/2604.24023/) (2026)
- [ChartSync: A Benchmark for Visuo-Logical Cascading Chart Editing](https://mlsanity.com/paper/2607.10301/) (2026)
- [UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark](https://mlsanity.com/paper/2512.17196/) (2025)
- [InEdit-Bench: Benchmarking Intermediate Logical Pathways for Intelligent Image Editing Models](https://mlsanity.com/paper/2603.03657/) (2026)
- [VTEdit-Bench: A Comprehensive Benchmark for Multi-Reference Image Editing Models in Virtual Try-On](https://mlsanity.com/paper/2603.11734/) (2026)
- [GIDE: Unlocking Diffusion LLMs for Precise Training-Free Image Editing](https://mlsanity.com/paper/2603.21176/) (2026)
- [Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generation](https://mlsanity.com/paper/2608.06751/) (2026)
- [Condition Weaving Meets Expert Modulation: Towards Universal and Controllable Image Generation](https://mlsanity.com/paper/2508.17364/) (2025)
- [BRIDGE: Background Routing and Isolated Discrete Gating for Coarse-Mask Local Editing](https://mlsanity.com/paper/2605.07846/) (2026)
- [Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications](https://mlsanity.com/paper/2607.25642/) (2026)
- [VETR: Execution-Grounded Self-Improving Agent for Symbolic Graphics Reasoning](https://mlsanity.com/paper/2610.13098/) (2026)

## Abstract (arXiv)

Image generation has witnessed significant advancements in the past few years. However, evaluating the performance of image generation models remains a formidable challenge. In this paper, we propose ICE-Bench, a unified and comprehensive benchmark designed to rigorously assess image generation models. Its comprehensiveness could be summarized in the following key features: (1) Coarse-to-Fine Tasks: We systematically deconstruct image generation into four task categories: No-ref/Ref Image Creating/Editing, based on the presence or absence of source images and reference images. And further decompose them into 31 fine-grained tasks covering a broad spectrum of image generation requirements, culminating in a comprehensive benchmark. (2) Multi-dimensional Metrics: The evaluation framework assesses image generation capabilities across 6 dimensions: aesthetic quality, imaging quality, prompt following, source consistency, reference consistency, and controllability. 11 metrics are introduced to support the multi-dimensional evaluation. Notably, we introduce VLLM-QA, an innovative metric designed to assess the success of image editing by leveraging large models. (3) Hybrid Data: The data comes from real scenes and virtual generation, which effectively improves data diversity and alleviates the bias problem in model evaluation. Through ICE-Bench, we conduct a thorough analysis of existing generation models, revealing both the challenging nature of our benchmark and the gap between current model capabilities and real-world generation requirements. To foster further advancements in the field, we will open-source ICE-Bench, including its dataset, evaluation code, and models, thereby providing a valuable resource for the research community.