# Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis

arXiv: https://arxiv.org/abs/2602.13028 · HTML: https://mlsanity.com/paper/2602.13028/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 13 Feb 2026.

## Builds on (most cited first, 20 of 20)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023)
- [EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods](https://mlsanity.com/paper/2310.02426/) (2023)
- [ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing](https://mlsanity.com/paper/2503.14482/) (2025)
- [GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions](https://mlsanity.com/paper/2310.15405/) (2023)
- [MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces](https://mlsanity.com/paper/2510.08783/) (2025)
- [Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts](https://mlsanity.com/paper/2505.17374/) (2025)
- [EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits](https://mlsanity.com/paper/2506.09988/) (2025)

## Built on by (most cited first, 8 of 8)

- [FORTIS: Benchmarking Over-Privilege in Agent Skills](https://mlsanity.com/paper/2605.09163/) (2026)
- [The Autonomy Tax: Defense Training Breaks LLM Agents](https://mlsanity.com/paper/2603.19423/) (2026)
- [MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation](https://mlsanity.com/paper/2605.28579/) (2026)
- [PaintBench: Deterministic Evaluation of Precise Visual Editing](https://mlsanity.com/paper/2606.00188/) (2026)
- [A Large-scale Evaluation of Text-guided Models for Facial Editing](https://mlsanity.com/paper/2608.28802/) (2026)
- [VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision](https://mlsanity.com/paper/2610.00666/) (2026)
- [Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation](https://mlsanity.com/paper/2610.01670/) (2026)
- [Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets](https://mlsanity.com/paper/2610.04381/) (2026)

## Abstract (arXiv)

Evaluating image editing models remains challenging due to the coarse granularity and limited interpretability of traditional metrics, which often fail to capture aspects important to human perception and intent. Such metrics frequently reward visually plausible outputs while overlooking controllability, edit localization, and faithfulness to user instructions. In this work, we introduce a fine-grained Multimodal Large Language Model (MLLM)-as-a-Judge framework for image editing that decomposes common evaluation notions into twelve fine-grained interpretable factors spanning image preservation, edit quality, and instruction fidelity. Building on this formulation, we present a new human-validated benchmark that integrates human judgments, MLLM-based evaluations, model outputs, and traditional metrics across diverse image editing tasks. Through extensive human studies, we show that the proposed MLLM judges align closely with human evaluations at a fine granularity, supporting their use as reliable and scalable evaluators. We further demonstrate that traditional image editing metrics are often poor proxies for these factors, failing to distinguish over-edited or semantically imprecise outputs, whereas our judges provide more intuitive and informative assessments in both offline and online settings. Together, this work introduces a benchmark, a principled factorization, and empirical evidence positioning fine-grained MLLM judges as a practical foundation for studying, comparing, and improving image editing approaches.