# Multistage Fine-tuning Strategies for Automatic Speech Recognition in Low-resource Languages

arXiv: https://arxiv.org/abs/2411.04573 · HTML: https://mlsanity.com/paper/2411.04573/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 8 of 8)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages](https://mlsanity.com/paper/2305.12606/) (2023)
- [What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations](https://mlsanity.com/paper/2409.02449/) (2024)
- [Leveraging supplementary text data to kick-start automatic speech recognition system development with limited transcriptions](https://mlsanity.com/paper/2302.04975/) (2023)

## Built on by (most cited first, 6 of 6)

- [Advancing STT for Low-Resource Real-World Speech](https://mlsanity.com/paper/2506.08836/) (2025)
- [WAXAL-NET: Finetuned Edge ASR Across 19 African Languages](https://mlsanity.com/paper/2606.02375/) (2026)
- [Fine-tuning Whisper for Pashto ASR: strategies and scale](https://mlsanity.com/paper/2604.06507/) (2026)
- [Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition](https://mlsanity.com/paper/2605.13087/) (2026)
- [From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2607.06289/) (2026)
- [Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptation](https://mlsanity.com/paper/2609.11302/) (2026)

## Abstract (arXiv)

This paper presents a novel multistage fine-tuning strategy designed to enhance automatic speech recognition (ASR) performance in low-resource languages using OpenAI's Whisper model. In this approach we aim to build ASR model for languages with limited digital resources by sequentially adapting the model across linguistically similar languages. We experimented this on the Malasar language, a Dravidian language spoken by approximately ten thousand people in the Western Ghats of South India. Malasar language faces critical challenges for technological intervention due to its lack of a native script and absence of digital or spoken data resources. Working in collaboration with Wycliffe India and Malasar community members, we created a spoken Malasar corpus paired with transcription in Tamil script, a closely related major language. In our approach to build ASR model for Malasar, we first build an intermediate Tamil ASR, leveraging higher data availability for Tamil annotated speech. This intermediate model is subsequently fine-tuned on Malasar data, allowing for more effective ASR adaptation despite limited resources. The multistage fine-tuning strategy demonstrated significant improvements over direct fine-tuning on Malasar data alone, achieving a word error rate (WER) of 51.9%, which is 4.5% absolute reduction when compared to the direct fine-tuning method. Further a WER reduction to 47.3% was achieved through punctuation removal in post-processing, which addresses formatting inconsistencies that impact evaluation. Our results underscore the effectiveness of sequential multistage fine-tuning combined with targeted post-processing as a scalable strategy for ASR system development in low-resource languages, especially where linguistic similarities can be leveraged to bridge gaps in training data.