mlsanityData to 14 Oct 2026

BhashaSetu: A Data-Centric Approach to Low-Resource Machine Translation

arXiv 2605.27050 ↗First posted 26 May 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We present BhashaSetu, a linguistically enriched English--Marathi parallel dataset addressing persistent data limitations in low-resource neural machine translation (NMT). Marathi, spoken by over 95 million people, remains underrepresented in high-quality parallel corpora across diverse domains. Our dataset comprises 2.78 million sentence pairs from heterogeneous sources including news, politics, healthcare, literature, and culture, with stemmed and lemmatized representations to support morphology-aware analysis. We benchmark multiple state-of-the-art translation models using BLEU, spBLEU, chrF++, and TER metrics, and conduct parameter-efficient fine-tuning of NLLB-200-distilled-600M using LoRA. A key finding from our ablation: corpus-level deduplication is the single largest preprocessing contributor to downstream quality (removing it reduces performance by 1.17 BLEU and 2.21 chrF++), demonstrating that disciplined cross-source corpus hygiene is a low-cost, high-impact intervention for low-resource, morphologically rich languages. The dataset is publicly released to promote reproducible and linguistically informed low-resource NMT research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 25 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

16 of 25 references.

References · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
  7. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  8. 2025WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialectscited by 79
  9. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  10. 2020PMIndia -- A Collection of Parallel Corpora of Languages of Indiacited by 42
  11. 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
  12. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
4 more
  1. 2020A Multilingual Parallel Corpora Collection Effort for Indian Languagescited by 16
  2. 2021Findings of the LoResMT 2021 Shared Task on COVID and Sign Language for Low-resource Languagescited by 10
  3. 2025Analysis of Indic Language Capabilities in LLMscited by 2
  4. 2024A Data Selection Approach for Enhancing Low Resource Machine Translation Using Cross-Lingual Sentence Representationscited by 1

Built on by

Not cited yet.