# Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization

arXiv: https://arxiv.org/abs/2412.19785 · HTML: https://mlsanity.com/paper/2412.19785/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 27 Dec 2024.

## Builds on (most cited first, 5 of 5)

- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [CLSRIL-23: Cross Lingual Speech Representations for Indic Languages](https://mlsanity.com/paper/2107.07402/) (2021)
- [Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text](https://mlsanity.com/paper/2408.05554/) (2024)

## Built on by (most cited first, 4 of 4)

- [Attention Is Not Always the Answer: Optimizing Voice Activity Detection with Simple Feature Fusion](https://mlsanity.com/paper/2506.01365/) (2025)
- [Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition](https://mlsanity.com/paper/2605.13087/) (2026)
- [Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languages](https://mlsanity.com/paper/2606.09535/) (2026)
- [Rethinking Indic AI from a Lens of Cultural Heritage Preservation](https://mlsanity.com/paper/2607.06544/) (2026)

## Abstract (arXiv)

Automatic speech recognition has recently seen a significant advancement with large foundational models such as Whisper. However, these models often struggle to perform well in low-resource languages, such as Indian languages. This paper explores two novel approaches to enhance Whisper's multilingual speech recognition performance in Indian languages. First, we propose prompt-tuning with language family information, which enhances Whisper's accuracy in linguistically similar languages. Second, we introduce a novel tokenizer that reduces the number of generated tokens, thereby accelerating Whisper's inference speed. Our extensive experiments demonstrate that the tokenizer significantly reduces inference time, while prompt-tuning enhances accuracy across various Whisper model sizes, including Small, Medium, and Large. Together, these techniques achieve a balance between optimal WER and inference speed.