mlsanityData to 7 Oct 2026

VIXEN: Visual Text Comparison Network for Image Difference Captioning

arXiv 2402.19119 ↗First posted 29 Feb 2024cs.CV cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

We present VIXEN - a technique that succinctly summarizes in text the visual differences between a pair of images in order to highlight any content manipulation present. Our proposed network linearly maps image features in a pairwise manner, constructing a soft prompt for a pretrained large language model. We address the challenge of low volume of training data and lack of manipulation variety in existing image difference captioning (IDC) datasets by training on synthetically manipulated images from the recent InstructPix2Pix dataset generated via prompt-to-prompt editing framework. We augment this dataset with change summaries produced via GPT-3. We show that VIXEN produces state-of-the-art, comprehensible difference captions for diverse image contents and edit types, offering a potential mitigation against misinformation disseminated via manipulated image content. Code and data are available at http://github.com/alexblck/vixen

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

40 of 55 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
28 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  3. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  7. 2016Self-critical Sequence Training for Image Captioningcited by 544
  8. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  9. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  10. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  11. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 231
  12. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  13. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 175
  14. 2022Linearly Mapping from Image to Text Spacecited by 139
  15. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  16. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  17. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  18. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  19. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  20. 2019Expressing Visual Relationships via Languagecited by 49
  21. 2019Hierarchy Parsing for Image Captioningcited by 44
  22. 2017Stack-Captioning: Coarse-to-Fine Learning for Image Captioningcited by 27
  23. 2020Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioningcited by 26
  24. 2022CLIP4IDC: CLIP for Image Difference Captioningcited by 25
  25. 2018The PS-Battles Dataset - an Image Collection for Image Manipulation Detectioncited by 16
  26. 2023Neighborhood Contrastive Transformer for Change Captioningcited by 16
  27. 2019Adaptively Aligned Image Captioning via Adaptive Attention Timecited by 15
  28. 2021OSCAR-Net: Object-centric Scene Graph Attention for Image Attributioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 1; 2026: 6.

12 papers.

  1. 2024Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Modelscited by 17
  2. 2024A Survey of Data Synthesis Approachescited by 8
  3. 2024Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioningcited by 6
  4. 2024TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Modelscited by 2
  5. 2025Composed Multi-modal Retrieval: A Survey of Approaches and Applicationscited by 2
  6. 2026Imagine How To Change: Explicit Procedure Modeling for Change Captioningcited by 1
  7. 2026DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?cited by 1
  8. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  9. 2026C3-Bench: A Context-Aware Change Captioning Benchmarkcited by 0
  10. 2026Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testingcited by 0
  11. 2026Different Changes Require Different Reasoning: Change-Type-Specialized Experts for Robust Change Captioningcited by 0
  12. 2026MinCU: A Fine-Grained Benchmark for Grounded Minimal-Change Understanding in Image Pairscited by 0