# I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing

arXiv: https://arxiv.org/abs/2408.14180 · HTML: https://mlsanity.com/paper/2408.14180/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 91 of its 111 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Builds on (most cited first, 30 of 91)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)

## Built on by (most cited first, 30 of 63)

- [ImgEdit: A Unified Image Editing Dataset and Benchmark](https://mlsanity.com/paper/2505.20275/) (2025)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models](https://mlsanity.com/paper/2505.16707/) (2025)
- [Judge Anything: MLLM as a Judge Across Any Modality](https://mlsanity.com/paper/2503.17489/) (2025)
- [ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependencies](https://mlsanity.com/paper/2506.12830/) (2025)
- [Factuality Matters: When Image Generation and Editing Meet Structured Visuals](https://mlsanity.com/paper/2510.05091/) (2025)
- [GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing](https://mlsanity.com/paper/2505.11493/) (2025)
- [CompBench: Benchmarking Complex Instruction-guided Image Editing](https://mlsanity.com/paper/2505.12200/) (2025)
- [PICABench: How Far Are We from Physically Realistic Image Editing?](https://mlsanity.com/paper/2510.17681/) (2025)
- [ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation](https://mlsanity.com/paper/2511.01163/) (2025)
- [EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing](https://mlsanity.com/paper/2509.13399/) (2025)
- [UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits](https://mlsanity.com/paper/2512.02790/) (2025)
- [MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation](https://mlsanity.com/paper/2511.22989/) (2025)
- [Image Editing with Diffusion Models: A Survey](https://mlsanity.com/paper/2504.13226/) (2025)
- [MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editing](https://mlsanity.com/paper/2506.01921/) (2025)
- [I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models](https://mlsanity.com/paper/2512.04660/) (2025)
- [ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks](https://mlsanity.com/paper/2603.27862/) (2026)
- [The Promise of RL for Autoregressive Image Editing](https://mlsanity.com/paper/2508.01119/) (2025)
- [Improving Editability in Image Generation with Layer-wise Memory](https://mlsanity.com/paper/2505.01079/) (2025)
- [Understanding Generative AI Capabilities in Everyday Image Editing Tasks](https://mlsanity.com/paper/2505.16181/) (2025)
- [Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework](https://mlsanity.com/paper/2606.19073/) (2026)
- [ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation](https://mlsanity.com/paper/2507.07317/) (2025)
- [EditHF-1M: A Million-Scale Rich Human Preference Feedback for Image Editing](https://mlsanity.com/paper/2603.14916/) (2026)
- [ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services](https://mlsanity.com/paper/2604.24023/) (2026)
- [ChartSync: A Benchmark for Visuo-Logical Cascading Chart Editing](https://mlsanity.com/paper/2607.10301/) (2026)
- [CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions](https://mlsanity.com/paper/2603.26174/) (2026)
- [CAMEO: A Conditional and Quality-Aware Multi-Agent Image Editing Orchestrator](https://mlsanity.com/paper/2604.03156/) (2026)
- [CPI-Bench: A Comprehensive, Practical and Intelligent Benchmark for Real-World Image Editing](https://mlsanity.com/paper/2608.14546/) (2026)
- [What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models](https://mlsanity.com/paper/2505.20405/) (2025)
- [AutoEdit: Automatic Hyperparameter Tuning for Image Editing](https://mlsanity.com/paper/2509.15031/) (2025)

## Abstract (arXiv)

Significant progress has been made in the field of Instruction-based Image Editing (IIE). However, evaluating these models poses a significant challenge. A crucial requirement in this field is the establishment of a comprehensive evaluation benchmark for accurately assessing editing results and providing valuable insights for its further development. In response to this need, we propose I2EBench, a comprehensive benchmark designed to automatically evaluate the quality of edited images produced by IIE models from multiple dimensions. I2EBench consists of 2,000+ images for editing, along with 4,000+ corresponding original and diverse instructions. It offers three distinctive characteristics: 1) Comprehensive Evaluation Dimensions: I2EBench comprises 16 evaluation dimensions that cover both high-level and low-level aspects, providing a comprehensive assessment of each IIE model. 2) Human Perception Alignment: To ensure the alignment of our benchmark with human perception, we conducted an extensive user study for each evaluation dimension. 3) Valuable Research Insights: By analyzing the advantages and disadvantages of existing IIE models across the 16 dimensions, we offer valuable research insights to guide future development in the field. We will open-source I2EBench, including all instructions, input images, human annotations, edited images from all evaluated methods, and a simple script for evaluating the results from new IIE models. The code, dataset and generated images from all IIE models are provided in github: https://github.com/cocoshe/I2EBench.