mlsanityData to 14 Oct 2026

Multilingual Representation Distillation with Contrastive Learning

arXiv 2210.05033 ↗First posted 10 Oct 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Multilingual sentence representations from large models encode semantic information from two or more languages and can be used for different cross-lingual information retrieval and matching tasks. In this paper, we integrate contrastive learning into multilingual representation distillation and use it for quality estimation of parallel sentences (i.e., find semantically similar sentences that can be used as translations of each other). We validate our approach with multilingual similarity search and corpus filtering tasks. Experiments across different low-resource languages show that our method greatly outperforms previous sentence encoders such as LASER, LASER3, and LaBSE.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

31 of 45 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2020Supervised Contrastive Learningcited by 2,829
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
19 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  4. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  5. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  6. 2019Contrastive Representation Distillationcited by 748
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  8. 2020Contrastive Learning with Hard Negative Samplescited by 628
  9. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  10. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  11. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  12. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  13. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  14. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  15. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  16. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  17. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 62
  18. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  19. 2022Bitext Mining for Low-Resource Languages via Contrastive Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 3; 2026: 1.

11 papers.

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  2. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  3. 2023Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillationcited by 6
  4. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  5. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  6. 2023Multilingual Sentence-Level Semantic Search using Meta-Distillation Learningcited by 1
  7. 2024Making Sentence Embeddings Robust to User-Generated Contentcited by 1
  8. 2024NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Modelscited by 1
  9. 2025LANGALIGN: Enhancing Non-English Language Models via Cross-Lingual Embedding Alignmentcited by 0
  10. 2025Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingualcited by 0
  11. 2026LLMs Get Smarter from Targeted Synthetic Multilingual Datacited by 0