# Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languages

arXiv: https://arxiv.org/abs/2411.04699 · HTML: https://mlsanity.com/paper/2411.04699/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 49 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 29 of 29)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [mSLAM: Massively multilingual joint pre-training for speech and text](https://mlsanity.com/paper/2202.01374/) (2022)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [Speech Translation and the End-to-End Promise: Taking Stock of Where We Are](https://mlsanity.com/paper/2004.06358/) (2020)
- [AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs](https://mlsanity.com/paper/2311.06753/) (2023)
- [Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://mlsanity.com/paper/2106.01045/) (2021)
- [IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages](https://mlsanity.com/paper/2403.01926/) (2024)
- [SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations](https://mlsanity.com/paper/2211.04508/) (2022)
- [Speech Translation with Large Language Models: An Industrial Practice](https://mlsanity.com/paper/2312.13585/) (2023)
- [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022)
- [Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder](https://mlsanity.com/paper/2010.13047/) (2020)
- [Robust Neural Machine Translation for Clean and Noisy Speech Transcripts](https://mlsanity.com/paper/1910.10238/) (2019)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [Breaking the Data Barrier: Towards Robust Speech Translation via Adversarial Stability Training](https://mlsanity.com/paper/1909.11430/) (2019)
- [indic-punct: An automatic punctuation restoration and inverse text normalization framework for Indic languages](https://mlsanity.com/paper/2203.16825/) (2022)
- [CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving](https://mlsanity.com/paper/2406.10993/) (2024)
- [Coursera Corpus Mining and Multistage Fine-Tuning for Improving Lectures Translation](https://mlsanity.com/paper/1912.11739/) (2019)
- [VAKTA-SETU: A Speech-to-Speech Machine Translation Service in Select Indic Languages](https://mlsanity.com/paper/2305.12518/) (2023)

## Built on by (most cited first, 5 of 5)

- [Parallel Corpora for Machine Translation in Low-resource Indic Languages: A Comprehensive Review](https://mlsanity.com/paper/2503.04797/) (2025)
- [UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages](https://mlsanity.com/paper/2509.21294/) (2025)
- [BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources](https://mlsanity.com/paper/2604.18423/) (2026)
- [CodeVaani: A Multilingual, Voice-Based Code Learning Assistant](https://mlsanity.com/paper/2511.20654/) (2025)
- [Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion](https://mlsanity.com/paper/2602.21646/) (2026)

## Abstract (arXiv)

Speech translation for Indian languages remains a challenging task due to the scarcity of large-scale, publicly available datasets that capture the linguistic diversity and domain coverage essential for real-world applications. Existing datasets cover a fraction of Indian languages and lack the breadth needed to train robust models that generalize beyond curated benchmarks. To bridge this gap, we introduce BhasaAnuvaad, the largest speech translation dataset for Indian languages, spanning over 44 thousand hours of audio and 17 million aligned text segments across 14 Indian languages and English. Our dataset is built through a threefold methodology: (a) aggregating high-quality existing sources, (b) large-scale web crawling to ensure linguistic and domain diversity, and (c) creating synthetic data to model real-world speech disfluencies. Leveraging BhasaAnuvaad, we train IndicSeamless, a state-of-the-art speech translation model for Indian languages that performs better than existing models. Our experiments demonstrate improvements in the translation quality, setting a new standard for Indian language speech translation. We will release all the code, data and model weights in the open-source, with permissive licenses to promote accessibility and collaboration.