# OneDiff: A Generalist Model for Image Difference Captioning

arXiv: https://arxiv.org/abs/2407.05645 · HTML: https://mlsanity.com/paper/2407.05645/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1402.1128/) (2014)
- [Expressing Visual Relationships via Language](https://mlsanity.com/paper/1906.07689/) (2019)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)
- [Neural Naturalist: Generating Fine-Grained Image Comparisons](https://mlsanity.com/paper/1909.04101/) (2019)
- [Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning](https://mlsanity.com/paper/2009.14352/) (2020)
- [CLIP4IDC: CLIP for Image Difference Captioning](https://mlsanity.com/paper/2206.00629/) (2022)
- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [L2C: Describing Visual Differences Needs Semantic Understanding of Individuals](https://mlsanity.com/paper/2102.01860/) (2021)

## Built on by (most cited first, 15 of 15)

- [CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Dataset](https://mlsanity.com/paper/2411.11360/) (2024)
- [OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning](https://mlsanity.com/paper/2503.11093/) (2025)
- [Finetuning CLIP to Reason about Pairwise Differences](https://mlsanity.com/paper/2409.09721/) (2024)
- [AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison](https://mlsanity.com/paper/2603.13779/) (2026)
- [VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?](https://mlsanity.com/paper/2603.07888/) (2026)
- [TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Models](https://mlsanity.com/paper/2412.18675/) (2024)
- [VerLM: Explaining Face Verification Using Natural Language](https://mlsanity.com/paper/2601.01798/) (2026)
- [Imagine How To Change: Explicit Procedure Modeling for Change Captioning](https://mlsanity.com/paper/2603.05969/) (2026)
- [DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning](https://mlsanity.com/paper/2605.04503/) (2026)
- [DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?](https://mlsanity.com/paper/2605.29615/) (2026)
- [Rethinking Infrastructure Inspection as Image Difference Classification: A Traffic Sign Case Study](https://mlsanity.com/paper/2606.06375/) (2026)
- [C3-Bench: A Context-Aware Change Captioning Benchmark](https://mlsanity.com/paper/2606.25445/) (2026)
- [DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues](https://mlsanity.com/paper/2606.26602/) (2026)
- [Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing](https://mlsanity.com/paper/2607.01728/) (2026)
- [MinCU: A Fine-Grained Benchmark for Grounded Minimal-Change Understanding in Image Pairs](https://mlsanity.com/paper/2609.23336/) (2026)

## Abstract (arXiv)

In computer vision, Image Difference Captioning (IDC) is crucial for accurately describing variations between closely related images. Traditional IDC methods often rely on specialist models, which restrict their applicability across varied contexts. This paper introduces the OneDiff model, a novel generalist approach that utilizes a robust vision-language model architecture, integrating a siamese image encoder with a Visual Delta Module. This innovative configuration allows for the precise detection and articulation of fine-grained differences between image pairs. OneDiff is trained through a dual-phase strategy, encompassing Coupled Sample Training and multi-task learning across a diverse array of data types, supported by our newly developed DiffCap Dataset. This dataset merges real-world and synthetic data, enhancing the training process and bolstering the model's robustness. Extensive testing on diverse IDC benchmarks, such as Spot-the-Diff, Image-Editing-Request, and Birds-to-Words, shows that OneDiff consistently outperforms existing state-of-the-art models in accuracy and adaptability, achieving improvements of up to 97% CIDEr points in average. By setting a new benchmark in IDC, OneDiff paves the way for more versatile and effective applications in detecting and describing visual differences. The code, models, and data will be made publicly available.