# OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning

arXiv: https://arxiv.org/abs/2503.11093 · HTML: https://mlsanity.com/paper/2503.11093/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 30 of 34)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models](https://mlsanity.com/paper/2404.13013/) (2024)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [Expressing Visual Relationships via Language](https://mlsanity.com/paper/1906.07689/) (2019)
- [Describing and Localizing Multiple Changes with Transformers](https://mlsanity.com/paper/2103.14146/) (2021)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)
- [Neural Naturalist: Generating Fine-Grained Image Comparisons](https://mlsanity.com/paper/1909.04101/) (2019)
- [Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning](https://mlsanity.com/paper/2009.14352/) (2020)
- [CLIP4IDC: CLIP for Image Difference Captioning](https://mlsanity.com/paper/2206.00629/) (2022)
- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance](https://mlsanity.com/paper/2407.14032/) (2024)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [OneDiff: A Generalist Model for Image Difference Captioning](https://mlsanity.com/paper/2407.05645/) (2024)

## Built on by (most cited first, 12 of 12)

- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison](https://mlsanity.com/paper/2603.13779/) (2026)
- [Embedding-perturbed Exploration Preference Optimization for Flow Models](https://mlsanity.com/paper/2605.15803/) (2026)
- [ViDiC: Video Difference Captioning](https://mlsanity.com/paper/2512.03405/) (2025)
- [Imagine How To Change: Explicit Procedure Modeling for Change Captioning](https://mlsanity.com/paper/2603.05969/) (2026)
- [DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning](https://mlsanity.com/paper/2605.04503/) (2026)
- [DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?](https://mlsanity.com/paper/2605.29615/) (2026)
- [DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues](https://mlsanity.com/paper/2606.26602/) (2026)
- [Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing](https://mlsanity.com/paper/2607.01728/) (2026)
- [VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification](https://mlsanity.com/paper/2609.06245/) (2026)
- [Spot-the-shift: Evaluating Grounded Image Difference Captioning of Long-term Changes](https://mlsanity.com/paper/2609.10356/) (2026)
- [MinCU: A Fine-Grained Benchmark for Grounded Minimal-Change Understanding in Image Pairs](https://mlsanity.com/paper/2609.23336/) (2026)

## Abstract (arXiv)

Image Difference Captioning (IDC) aims to generate natural language descriptions of subtle differences between image pairs, requiring both precise visual change localization and coherent semantic expression. Despite recent advancements, existing datasets often lack breadth and depth, limiting their applicability in complex and dynamic environments: (1) from a breadth perspective, current datasets are constrained to limited variations of objects in specific scenes, and (2) from a depth perspective, prior benchmarks often provide overly simplistic descriptions. To address these challenges, we introduce OmniDiff, a comprehensive dataset comprising 324 diverse scenarios-spanning real-world complex environments and 3D synthetic settings-with fine-grained human annotations averaging 60 words in length and covering 12 distinct change types. Building on this foundation, we propose M3Diff, a MultiModal large language model enhanced by a plug-and-play Multi-scale Differential Perception (MDP) module. This module improves the model's ability to accurately identify and describe inter-image differences while maintaining the foundational model's generalization capabilities. With the addition of the OmniDiff dataset, M3Diff achieves state-of-the-art performance across multiple benchmarks, including Spot-the-Diff, IEdit, CLEVR-Change, CLEVR-DC, and OmniDiff, demonstrating significant improvements in cross-scenario difference recognition accuracy compared to existing methods. The dataset, code, and models will be made publicly available to support further research.