# Aligning Information Capacity Between Vision and Language via Dense-to-Sparse Feature Distillation for Image-Text Matching

arXiv: https://arxiv.org/abs/2503.14953 · HTML: https://mlsanity.com/paper/2503.14953/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Learning the Best Pooling Strategy for Visual Semantic Embedding](https://mlsanity.com/paper/2011.04305/) (2020)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval](https://mlsanity.com/paper/2003.03772/) (2020)
- [CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval](https://mlsanity.com/paper/1909.05506/) (2019)
- [Graph Structured Network for Image-Text Matching](https://mlsanity.com/paper/2004.00277/) (2020)
- [Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1906.04402/) (2019)
- [DreamLIP: Language-Image Pre-training with Long Captions](https://mlsanity.com/paper/2403.17007/) (2024)
- [Learning Semantic Concepts and Order for Image and Sentence Matching](https://mlsanity.com/paper/1712.02036/) (2017)
- [Consensus-Aware Visual-Semantic Embedding for Image-Text Matching](https://mlsanity.com/paper/2007.08883/) (2020)
- [Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models](https://mlsanity.com/paper/1711.06420/) (2017)
- [Improved Probabilistic Image-Text Representations](https://mlsanity.com/paper/2305.18171/) (2023)
- [Universal Weighting Metric Learning for Cross-Modal Matching](https://mlsanity.com/paper/2010.03403/) (2020)
- [Improving Cross-Modal Retrieval with Set of Diverse Embeddings](https://mlsanity.com/paper/2211.16761/) (2022)
- [LoTLIP: Improving Language-Image Pre-training for Long Text Understanding](https://mlsanity.com/paper/2410.05249/) (2024)
- [Cross-modal Active Complementary Learning with Self-refining Correspondence](https://mlsanity.com/paper/2310.17468/) (2023)
- [Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment](https://mlsanity.com/paper/2503.06974/) (2025)

## Built on by (most cited first, 5 of 5)

- [SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment](https://mlsanity.com/paper/2511.01390/) (2025)
- [Camouflage-aware Image-Text Retrieval via Expert Collaboration](https://mlsanity.com/paper/2604.01251/) (2026)
- [GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective](https://mlsanity.com/paper/2605.15723/) (2026)
- [Intra-Modal Neighbors Never Lie: Rectifying Inter-Modal Noisy Correspondence via Graph-Based Intra-Modal Reasoning](https://mlsanity.com/paper/2606.04061/) (2026)
- [AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization](https://mlsanity.com/paper/2607.13805/) (2026)

## Abstract (arXiv)

Enabling Visual Semantic Models to effectively handle multi-view description matching has been a longstanding challenge. Existing methods typically learn a set of embeddings to find the optimal match for each view's text and compute similarity. However, the visual and text embeddings learned through these approaches have limited information capacity and are prone to interference from locally similar negative samples. To address this issue, we argue that the information capacity of embeddings is crucial and propose Dense-to-Sparse Feature Distilled Visual Semantic Embedding (D2S-VSE), which enhances the information capacity of sparse text by leveraging dense text distillation. Specifically, D2S-VSE is a two-stage framework. In the pre-training stage, we align images with dense text to enhance the information capacity of visual semantic embeddings. In the fine-tuning stage, we optimize two tasks simultaneously, distilling dense text embeddings to sparse text embeddings while aligning images and sparse texts, enhancing the information capacity of sparse text embeddings. Our proposed D2S-VSE model is extensively evaluated on the large-scale MS-COCO and Flickr30K datasets, demonstrating its superiority over recent state-of-the-art methods.