mlsanityData to 14 Oct 2026

Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages

arXiv 2205.12654 ↗First posted 25 May 2022cs.CLcited by 48

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Scaling multilingual representation learning beyond the hundred most frequent languages is challenging, in particular to cover the long tail of low-resource languages. A promising approach has been to train one-for-all multilingual models capable of cross-lingual transfer, but these models often suffer from insufficient capacity and interference between unrelated languages. Instead, we move away from this approach and focus on training multiple language (family) specific representations, but most prominently enable all languages to still be encoded in the same representational space. To achieve this, we focus on teacher-student training, allowing all encoders to be mutually compatible for bitext mining, and enabling fast learning of new languages. We introduce a new teacher-student training scheme which combines supervised and self-supervised training, allowing encoders to take advantage of monolingual training data, which is valuable in the low-resource setting. Our approach significantly outperforms the original LASER encoder. We study very low-resource languages and handle 50 African languages, many of which are not covered by any other model. For these languages, we train sentence encoders, mine bitexts, and validate the bitexts by training NMT systems.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 31 of its 39 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

31 of 39 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  5. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  6. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  7. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  8. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  11. 2018Learning Word Vectors for 157 Languagescited by 280
  12. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
19 more
  1. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  2. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  3. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  4. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  5. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
  6. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  7. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  8. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  9. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  10. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  11. 2018Effective Parallel Corpus Mining using Bilingual Sentence Embeddingscited by 39
  12. 2018Filtering and Mining Parallel Data in a Joint Multilingual Spacecited by 32
  13. 2020Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoderscited by 23
  14. 2022Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learningcited by 19
  15. 2021AI4D -- African Language Programcited by 18
  16. 2020FFR v1.1: Fon-French Neural Machine Translationcited by 16
  17. 2019Benchmarking Neural Machine Translation for Southern African Languagescited by 9
  18. 2021NLP for Ghanaian Languagescited by 5
  19. 2021English2Gbe: A multilingual machine translation model for {Fon/Ewe}Gbecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 16; 2024: 8; 2025: 7; 2026: 7.

48 papers.

Sort:

By area:

  1. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  2. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  3. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  4. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  5. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  6. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  7. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 36
  8. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  9. 2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
  10. 2023Revisiting non-English Text Simplification: A Unified Multilingual Benchmarkcited by 27
  11. 2023Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scalecited by 24
  12. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
36 more
  1. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  2. 2023L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERTcited by 15
  3. 2022Multilingual Representation Distillation with Contrastive Learningcited by 11
  4. 2023Language Models are Universal Embedderscited by 11
  5. 2022L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathicited by 10
  6. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  7. 2024Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruningcited by 10
  8. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  9. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  10. 2022Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translationcited by 8
  11. 2023Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translationcited by 8
  12. 2023Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularizationcited by 7
  13. 2024PersianMind: A Cross-Lingual Persian-English Large Language Modelcited by 7
  14. 2022You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Modelscited by 6
  15. 2022Very Low Resource Sentence Alignment: Luhya and Swahilicited by 6
  16. 2024Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimizationcited by 5
  17. 2024MEXMA: Token-level objectives improve sentence representationscited by 4
  18. 2023xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languagescited by 3
  19. 2025Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarioscited by 3
  20. 2023Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfercited by 2
  21. 2024Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignmentcited by 2
  22. 2022EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learningcited by 1
  23. 2023Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings?cited by 1
  24. 2024Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfercited by 1
  25. 2026TurkicNLP: An NLP Toolkit for Turkic Languagescited by 1
  26. 2023Is a Prestigious Job the same as a Prestigious Country? A Case Study on Multilingual Sentence Embeddings and European Countriescited by 0
  27. 2023Leveraging Closed-Access Multilingual Embedding for Automatic Sentence Alignment in Low Resource Languagescited by 0
  28. 2025WolBanking77: Wolof Banking Speech Intent Classification Datasetcited by 0
  29. 2025Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddingscited by 0
  30. 2025How Good is BLI as an Alignment Measure: A Study in Word Embedding Paradigmcited by 0
  31. 2026BYOL: Bring Your Own Language Into LLMscited by 0
  32. 2026Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Modelscited by 0
  33. 2026Unified Vision-Language Modeling via Concept Space Alignmentcited by 0
  34. 2026FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddingscited by 0
  35. 2026Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Contentcited by 0
  36. 2026Trilingual Topic Modeling of Sri Lankan Parliamentary Debatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining
  2. 2021MergeDistill: Merging Pre-trained Language Models using Distillation
  3. 2024MINERS: Multilingual Language Models as Semantic Retrievers
  4. 2026Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
  5. 2025TiME: Tiny Monolingual Encoders for Efficient NLP Pipelines
  6. 2025On Multilingual Encoder Language Model Compression for Low-Resource Languages
  7. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World
  8. 2025Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders