mlsanityData to 14 Oct 2026

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

arXiv 2210.03625 ↗First posted 7 Oct 2022cs.CL cs.CV cs.MMcited by 3

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Multilingual text-video retrieval methods have improved significantly in recent years, but the performance for other languages lags behind English. We propose a Cross-Lingual Cross-Modal Knowledge Distillation method to improve multilingual text-video retrieval. Inspired by the fact that English text-video retrieval outperforms other languages, we train a student model using input text in different languages to match the cross-modal predictions from teacher models using input text in English. We propose a cross entropy based objective which forces the distribution over the student's text-video similarity scores to be similar to those of the teacher models. We introduce a new multilingual video dataset, Multi-YouCook2, by translating the English captions in the YouCook2 video dataset to 8 other languages. Our method improves multilingual text-video retrieval performance on Multi-YouCook2 and several other datasets such as Multi-MSRVTT and VATEX. We also conducted an analysis on the effectiveness of different multilingual text models as teachers. The code, models, and dataset are available at https://github.com/roudimit/c2kd.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 36 of its 49 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

36 of 49 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2017The Kinetics Human Action Video Datasetcited by 1,909
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
24 more
  1. 2016Language Modeling with Gated Convolutional Networkscited by 702
  2. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  3. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  4. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  5. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  6. 2016Exploring the Limits of Language Modelingcited by 173
  7. 2020Support-set bottlenecks for video-text representation learningcited by 170
  8. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  9. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 111
  10. 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
  11. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  12. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  13. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  14. 2017Learnable pooling with Context Gating for video classificationcited by 58
  15. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  16. 2021Cross-Modal Discrete Representation Learningcited by 35
  17. 2020Visual Grounding in Video for Unsupervised Word Translationcited by 27
  18. 2021GEM: A General Evaluation Benchmark for Multimodal Taskscited by 16
  19. 2021Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillationcited by 13
  20. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classificationcited by 12
  21. 2021MTVR: Multilingual Moment Retrieval in Videoscited by 8
  22. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  23. 20223MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videoscited by 8
  24. 2022cViL: Cross-Lingual Training of Vision-Language Models using Knowledge Distillationcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 0; 2025: 1.

3 papers.

  1. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  2. 2023MultiVENT: Multilingual Videos of Events with Aligned Natural Textcited by 7
  3. 2025Leveraging Auxiliary Information in Text-to-Video Retrieval: A Reviewcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
  2. 2025Q2E: Query-to-Event Decomposition for Zero-Shot Multilingual Text-to-Video Retrieval
  3. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation
  4. 2021Learning Cross-Lingual IR from an English Retriever
  5. 2024Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval
  6. 2023WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge
  7. 2022Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching
  8. 2021VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer