mlsanityData to 14 Oct 2026

DiffCLIP: Differential Attention Meets CLIP

arXiv 2503.06626 ↗First posted 9 Mar 2025cs.CV cs.AI cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We propose DiffCLIP, a novel vision-language model that extends the differential attention mechanism to CLIP architectures. Differential attention was originally developed for large language models to amplify relevant context while canceling out noisy information. In this work, we integrate this mechanism into CLIP's dual encoder (image and text) framework. With minimal additional parameters, DiffCLIP achieves superior performance on image-text understanding tasks. Across zero-shot classification, retrieval, and robustness benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably, these gains come with negligible computational overhead, demonstrating that differential attention can significantly enhance multi-modal representations without sacrificing efficiency. Code can be found at https://github.com/hammoudhasan/DiffCLIP.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 52 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

40 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions25 Nov
  2. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captions11 Oct
  3. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining29 Apr

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
25 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  5. 2019Natural Adversarial Examplescited by 1,408
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  9. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  10. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  11. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  12. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  13. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  14. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  15. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  16. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  17. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  18. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  19. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  20. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 55
  21. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  22. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  23. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  24. 2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17
  25. 2024Efficient Vision-Language Pre-training by Cluster Maskingcited by 13

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025SuperCLIP: CLIP with Simple Classification Supervisioncited by 4
  2. 2025Understanding Sensitivity of Differential Attention through the Lens of Adversarial Robustnesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
  2. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  3. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
  4. 2025CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation
  5. 2025Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
  6. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study
  7. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
  8. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning