# Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation

arXiv: https://arxiv.org/abs/2412.15939 · HTML: https://mlsanity.com/paper/2412.15939/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Expressing Visual Relationships via Language](https://mlsanity.com/paper/1906.07689/) (2019)
- [Describing and Localizing Multiple Changes with Transformers](https://mlsanity.com/paper/2103.14146/) (2021)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)
- [Progressive Scale-aware Network for Remote sensing Image Change Captioning](https://mlsanity.com/paper/2303.00355/) (2023)
- [Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning](https://mlsanity.com/paper/2009.14352/) (2020)

## Built on by (most cited first, 3 of 3)

- [Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning](https://mlsanity.com/paper/2509.18369/) (2025)
- [Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing](https://mlsanity.com/paper/2607.01728/) (2026)
- [Different Changes Require Different Reasoning: Change-Type-Specialized Experts for Robust Change Captioning](https://mlsanity.com/paper/2609.01136/) (2026)

## Abstract (arXiv)

The rise of the generative models quality during the past years enabled the generation of edited variations of images at an important scale. To counter the harmful effects of such technology, the Image Difference Captioning (IDC) task aims to describe the differences between two images. While this task is successfully handled for simple 3D rendered images, it struggles on real-world images. The reason is twofold: the training data-scarcity, and the difficulty to capture fine-grained differences between complex images. To address those issues, we propose in this paper a simple yet effective framework to both adapt existing image captioning models to the IDC task and augment IDC datasets. We introduce BLIP2IDC, an adaptation of BLIP2 to the IDC task at low computational cost, and show it outperforms two-streams approaches by a significant margin on real-world IDC datasets. We also propose to use synthetic augmentation to improve the performance of IDC models in an agnostic fashion. We show that our synthetic augmentation strategy provides high quality data, leading to a challenging new dataset well-suited for IDC named Syned1.