mlsanityData to 14 Oct 2026

WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

arXiv 1907.05791 ↗First posted 10 Jul 2019cs.CLcited by 223

Not on the map. Why?

Abstract (arXiv)

We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 85 languages, including several dialects or low-resource languages. We do not limit the the extraction process to alignments with English, but systematically consider all possible language pairs. In total, we are able to extract 135M parallel sentences for 1620 different language pairs, out of which only 34M are aligned with English. This corpus of parallel sentences is freely available at https://github.com/facebookresearch/LASER/tree/master/tasks/WikiMatrix. To get an indication on the quality of the extracted bitexts, we train neural MT baseline systems on the mined data only for 1886 languages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs. The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English.

Key facts. Cited by 223 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 53; 2022: 58; 2023: 30; 2024: 41; 2025: 19; 2026: 22.

223 papers.

Sort:

By area:

  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  2. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  3. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  4. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  5. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  6. 2022Relative representations enable zero-shot latent space communicationcited by 191
  7. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  8. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  9. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
  10. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  11. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  12. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 109
38 more
  1. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  2. 2021Survey of Low-Resource Machine Translationcited by 92
  3. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  4. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  5. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  6. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  7. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  8. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  9. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  10. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 56
  11. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  12. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  13. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  14. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  15. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  16. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 39
  17. 2021The Effect of Domain and Diacritics in Yor`ub'a-English Neural Machine Translationcited by 38
  18. 2022A Survey on Cross-Lingual Summarizationcited by 38
  19. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  20. 2023ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languagescited by 34
  21. 2022Frustratingly Easy Label Projection for Cross-lingual Transfercited by 33
  22. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Modelcited by 33
  23. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  24. 2021Many-to-English Machine Translation Tools, Data, and Pretrained Modelscited by 31
  25. 2022Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russiancited by 29
  26. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29
  27. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  28. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  29. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  30. 2023Escaping the sentence-level paradigm in machine translationcited by 27
  31. 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
  32. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  33. 2022Human Interpretation of Saliency-based Explanation Over Textcited by 25
  34. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  35. 2023GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generationcited by 25
  36. 2022Efficient Cluster-Based k-Nearest-Neighbor Machine Translationcited by 24
  37. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  38. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24

50 most cited of 223 All 223 (JSON)