mlsanityData to 7 Oct 2026

Distilling Vision-Language Pretraining for Efficient Cross-Modal Retrieval

arXiv 2405.14726 ↗First posted 23 May 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

``Learning to hash'' is a practical solution for efficient retrieval, offering fast search speed and low storage cost. It is widely applied in various applications, such as image-text cross-modal search. In this paper, we explore the potential of enhancing the performance of learning to hash with the proliferation of powerful large pre-trained models, such as Vision-Language Pre-training (VLP) models. We introduce a novel method named Distillation for Cross-Modal Quantization (DCMQ), which leverages the rich semantic knowledge of VLP models to improve hash representation learning. Specifically, we use the VLP as a `teacher' to distill knowledge into a `student' hashing model equipped with codebooks. This process involves the replacement of supervised labels, which are composed of multi-hot vectors and lack semantics, with the rich semantics of VLP. In the end, we apply a transformation termed Normalization with Paired Consistency (NPC) to achieve a discriminative target for distillation. Further, we introduce a new quantization method, Product Quantization with Gumbel (PQG) that promotes balanced codebook learning, thereby improving the retrieval performance. Extensive benchmark testing demonstrates that DCMQ consistently outperforms existing supervised cross-modal hashing approaches, showcasing its significant potential.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 62 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

36 of 62 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2020Knowledge Distillation: A Surveycited by 1,578
  11. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  12. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
24 more
  1. 2016A Survey on Learning to Hashcited by 119
  2. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  3. 2019Central Similarity Quantization for Efficient Image and Video Retrievalcited by 77
  4. 2022Multi-modal Alignment using Representation Codebookcited by 63
  5. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  6. 2016Deep Cross-Modal Hashingcited by 54
  7. 2017Deep Supervised Discrete Hashingcited by 36
  8. 2019Differentiable Product Quantization for End-to-End Embedding Compressioncited by 36
  9. 2018Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrievalcited by 29
  10. 2021Self-supervised Product Quantization for Deep Unsupervised Image Retrievalcited by 28
  11. 2019Triplet-Based Deep Hashing Network for Cross-Modal Retrievalcited by 26
  12. 2020Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashingcited by 26
  13. 2021KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillationcited by 23
  14. 2020Generalized Product Quantization Network for Semi-supervised Image Retrievalcited by 20
  15. 2017End-to-End Supervised Product Quantization for Image Search and Retrievalcited by 16
  16. 2017Unsupervised Generative Adversarial Cross-modal Hashingcited by 15
  17. 2020Comprehensive Graph-conditional Similarity Preserving Network for Unsupervised Cross-modal Hashingcited by 15
  18. 2021Deep Hash Distillation for Image Retrievalcited by 9
  19. 2015Composite Correlation Quantization for Efficient Multimodal Retrievalcited by 6
  20. 2019Ranking-based Deep Cross-modal Hashingcited by 6
  21. 2019Beyond Product Quantization: Deep Progressive Quantization for Image Retrievalcited by 5
  22. 2022Adaptive Structural Similarity Preserving for Unsupervised Cross Modal Hashingcited by 5
  23. 2019Collaborative Quantization for Cross-Modal Similarity Searchcited by 3
  24. 2021Refining BERT Embeddings for Document Hashing via Mutual Information Maximizationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting
  2. 2026Unsupervised Data-Efficient Cross-Modal Retrieval with Global-Neighborhood Alignment Hashing
  3. 2026Attribute-Prompted Kernel Hashing for Unsupervised Data-Efficient Cross-Modal Retrieval
  4. 2024Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models
  5. 2022COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval
  6. 2023DLIP: Distilling Language-Image Pre-training
  7. 2025PromptHash: Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrieval
  8. 2022Dynamic Contrastive Distillation for Image-Text Retrieval