mlsanityData to 7 Oct 2026

Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

arXiv 2104.05596 ↗First posted 12 Apr 2021cs.CLcited by 133

Not on the map. Why?

Abstract (arXiv)

We present Samanantar, the largest publicly available parallel corpora collection for Indic languages. The collection contains a total of 49.7 million sentence pairs between English and 11 Indic languages (from two language families). Specifically, we compile 12.4 million sentence pairs from existing, publicly-available parallel corpora, and additionally mine 37.4 million sentence pairs from the web, resulting in a 4x increase. We mine the parallel sentences from the web by combining many corpora, tools, and methods: (a) web-crawled monolingual corpora, (b) document OCR for extracting sentences from scanned documents, (c) multilingual representation models for aligning sentences, and (d) approximate nearest neighbor search for searching in a large collection of sentences. Human evaluation of samples from the newly mined corpora validate the high quality of the parallel sentences across 11 languages. Further, we extract 83.4 million sentence pairs between all 55 Indic language pairs from the English-centric parallel corpus using English as the pivot language. We trained multilingual NMT models spanning all these languages on Samanantar, which outperform existing models and baselines on publicly available benchmarks, such as FLORES, establishing the utility of Samanantar. Our data and models are available publicly at https://ai4bharat.iitm.ac.in/samanantar and we hope they will help advance research in NMT and multilingual NLP for Indic languages.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 43 of its 77 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

43 of 77 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,088
31 more
  1. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  2. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  3. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  4. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  5. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  6. 2017Six Challenges for Neural Machine Translationcited by 334
  7. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  8. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  9. 2019Massively Multilingual Neural Machine Translationcited by 246
  10. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  11. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  12. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  13. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  14. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  15. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  16. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  17. 2020TICO-19: the Translation Initiative for Covid-19cited by 67
  18. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 61
  19. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  20. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  21. 2020Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translationcited by 53
  22. 2018Trivial Transfer Learning for Low-Resource Neural Machine Translationcited by 47
  23. 2019Multilingual Neural Machine Translation with Language Clusteringcited by 47
  24. 2018Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corporacited by 40
  25. 2020PMIndia -- A Collection of Parallel Corpora of Languages of Indiacited by 40
  26. 2020Complete Multilingual Neural Machine Translationcited by 30
  27. 2020A Multilingual Parallel Corpora Collection Effort for Indian Languagescited by 16
  28. 2020Data Augmentation and Terminology Integration for Domain-Specific Sinhala-English-Tamil Statistical Machine Translationcited by 16
  29. 2020Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translationcited by 13
  30. 2020Revisiting Low Resource Status of Indian Languages in Machine Translationcited by 12
  31. 2020Neural Machine Translation System of Indic Languages -- An Attention based Approachcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 28; 2023: 29; 2024: 19; 2025: 23; 2026: 21.

133 papers.

Sort:

By area:

  1. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  4. 2021Survey of Low-Resource Machine Translationcited by 92
  5. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 67
  6. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 55
  7. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  8. 2021Towards Building ASR Systems for the Next Billion Userscited by 45
  9. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  10. 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
  11. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  12. 2022Using natural language prompts for machine translationcited by 27
38 more
  1. 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
  2. 2022IndicXNLI: Evaluating Multilingual Inference for Indian Languagescited by 26
  3. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  4. 2022A Survey of Adversarial Defences and Robustness in NLPcited by 22
  5. 2021Few-shot Controllable Style Transfer for Low-Resource Multilingual Settingscited by 20
  6. 2022Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languagescited by 20
  7. 2024Egalitarian Language Representation in Language Models: It All Begins with Tokenizerscited by 20
  8. 2022Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Userscited by 19
  9. 2022Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languagescited by 16
  10. 2022BanglaParaphrase: A High-Quality Bangla Paraphrase Datasetcited by 15
  11. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  12. 2023CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translationcited by 12
  13. 2023Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Setcited by 12
  14. 2023MILDSum: A Novel Benchmark Dataset for Multilingual Summarization of Indian Legal Case Judgmentscited by 10
  15. 2024Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruningcited by 10
  16. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  17. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  18. 2024From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentencescited by 8
  19. 2023Improving Machine Translation with Phrase Pair Injection and Corpus Filteringcited by 7
  20. 2024Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMscited by 7
  21. 2024Table Question Answering for Low-resourced Indic Languagescited by 7
  22. 2022MuCoT: Multilingual Contrastive Training for Question-Answering in Low-resource Languagescited by 6
  23. 2022Improving Multilingual Neural Machine Translation System for Indic Languagescited by 6
  24. 2024Pretraining Language Models Using Translationesecited by 6
  25. 2024A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual LLMscited by 6
  26. 2024Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Modelscited by 6
  27. 2025From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corporacited by 6
  28. 2025Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challengescited by 6
  29. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5
  30. 2023An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Modelscited by 5
  31. 2023IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languagescited by 5
  32. 2025PARAM-1 BharatGen 2.9B Modelcited by 5
  33. 2021The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understandingcited by 4
  34. 2021Attentive fine-tuning of Transformers for Translation of low-resourced languages @LoResMT 2021cited by 4
  35. 2022XAlign: Cross-lingual Fact-to-Text Alignment and Generation for Low-Resource Languagescited by 4
  36. 2022DICTDIS: Dictionary Constrained Disambiguation for Improved NMTcited by 4
  37. 2023Model Adaptation for ASR in low-resource Indian Languagescited by 4
  38. 2023Breaking Language Barriers: A Question Answering Dataset for Hindi and Marathicited by 4

50 most cited of 133 All 133 (JSON)