# Fine-tuning Whisper on Low-Resource Languages for Real-World Applications

arXiv: https://arxiv.org/abs/2412.15726 · HTML: https://mlsanity.com/paper/2412.15726/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 22 of 22)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Low-Memory Neural Network Training: A Technical Report](https://mlsanity.com/paper/1904.10631/) (2019)
- [SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German](https://mlsanity.com/paper/2103.11401/) (2021)
- [SDS-200: A Swiss German Speech to Standard German Text Corpus](https://mlsanity.com/paper/2205.09501/) (2022)
- [STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions](https://mlsanity.com/paper/2305.18855/) (2023)
- [Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts](https://mlsanity.com/paper/2311.01070/) (2023)
- [Swiss Parliaments Corpus, an Automatically Aligned Swiss German Speech to Standard German Text Corpus](https://mlsanity.com/paper/2010.02810/) (2020)
- [Learn and Don't Forget: Adding a New Language to ASR Foundation Models](https://mlsanity.com/paper/2407.06800/) (2024)
- [Adapting an ASR Foundation Model for Spoken Language Assessment](https://mlsanity.com/paper/2307.09378/) (2023)
- [SubER: A Metric for Automatic Evaluation of Subtitle Quality](https://mlsanity.com/paper/2205.05805/) (2022)
- [Spaiche: Extending State-of-the-Art ASR Models to Swiss German Dialects](https://mlsanity.com/paper/2304.11075/) (2023)
- [Swiss German Speech to Text system evaluation](https://mlsanity.com/paper/2207.00412/) (2022)
- [Dialect Transfer for Swiss German Speech Translation](https://mlsanity.com/paper/2310.09088/) (2023)
- [Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition](https://mlsanity.com/paper/2409.16954/) (2024)
- [Using fine-tuning and min lookahead beam search to improve Whisper](https://mlsanity.com/paper/2309.10299/) (2023)

## Built on by (most cited first, 11 of 11)

- [A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data](https://mlsanity.com/paper/2506.11130/) (2025)
- [Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages](https://mlsanity.com/paper/2503.23542/) (2025)
- [Scaling and Prompting for Improved End-to-End Spoken Grammatical Error Correction](https://mlsanity.com/paper/2505.21137/) (2025)
- [Advancing STT for Low-Resource Real-World Speech](https://mlsanity.com/paper/2506.08836/) (2025)
- [End-to-End Spoken Grammatical Error Correction](https://mlsanity.com/paper/2506.18532/) (2025)
- [Structured Sparsity and Weight-adaptive Pruning for Memory and Compute efficient Whisper models](https://mlsanity.com/paper/2510.12666/) (2025)
- [Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)](https://mlsanity.com/paper/2606.07608/) (2026)
- [TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech](https://mlsanity.com/paper/2606.30543/) (2026)
- [Voice Adaptation for Swiss German](https://mlsanity.com/paper/2505.22054/) (2025)
- [WARDEN: Endangered Indigenous Language Transcription and Translation with 6 Hours of Training Data](https://mlsanity.com/paper/2605.13846/) (2026)
- [Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models](https://mlsanity.com/paper/2607.17164/) (2026)

## Abstract (arXiv)

This paper presents a new approach to fine-tuning OpenAI's Whisper model for low-resource languages by introducing a novel data generation method that converts sentence-level data into a long-form corpus, using Swiss German as a case study. Non-sentence-level data, which could improve the performance of long-form audio, is difficult to obtain and often restricted by copyright laws. Our method bridges this gap by transforming more accessible sentence-level data into a format that preserves the model's ability to handle long-form audio and perform segmentation without requiring non-sentence-level data. Our data generation process improves performance in several real-world applications and leads to the development of a new state-of-the-art speech-to-text (STT) model for Swiss German. We compare our model with a non-fine-tuned Whisper and our previous state-of-the-art Swiss German STT models, where our new model achieves higher BLEU scores. Our results also indicate that the proposed method is adaptable to other low-resource languages, supported by written guidance and code that allows the creation of fine-tuned Whisper models, which keep segmentation capabilities and allow the transcription of longer audio files using only sentence-level data with high quality.