mlsanityData to 14 Oct 2026

MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval

arXiv 2310.19654 ↗First posted 30 Oct 2023cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Due to the success of large-scale visual-language pretraining (VLP) models and the widespread use of image-text retrieval in industry areas, it is now critically necessary to reduce the model size and streamline their mobile-device deployment. Single- and dual-stream model structures are commonly used in image-text retrieval with the goal of closing the semantic gap between textual and visual modalities. While single-stream models use deep feature fusion to achieve more accurate cross-model alignment, dual-stream models are better at offline indexing and fast inference.We propose a Multi-teacher Cross-modality Alignment Distillation (MCAD) technique to integrate the advantages of single- and dual-stream models. By incorporating the fused single-stream features into the image and text features of the dual-stream model, we formulate new modified teacher similarity distributions and features. Then, we conduct both distribution and feature distillation to boost the capability of the student dual-stream model, achieving high retrieval performance without increasing inference complexity.Extensive experiments demonstrate the remarkable performance and high efficiency of MCAD on image-text retrieval tasks. Furthermore, we implement a lightweight CLIP model on Snapdragon/Dimensity chips with only ~100M running memory and ~8.0ms search latency, achieving the mobile-device application of VLP models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

34 of 54 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2015Fast R-CNNcited by 2,933
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
22 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  10. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  11. 2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
  12. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  13. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  14. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 86
  15. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  16. 2019Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matchingcited by 49
  17. 2022MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Featurescited by 44
  18. 2019Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering Systemcited by 43
  19. 2021Distilled Dual-Encoder Model for Vision-Language Understandingcited by 22
  20. 2022Dynamic Contrastive Distillation for Image-Text Retrievalcited by 14
  21. 2022LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrievalcited by 9
  22. 2022Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrievalcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models
  2. 2022COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval
  3. 2024How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?
  4. 2025Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation
  5. 2024Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrieval
  6. 2021Cross-Modal Retrieval Augmentation for Multi-Modal Classification
  7. 2023MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval
  8. 2025Multimodal Representation Alignment for Cross-modal Information Retrieval