mlsanityData to 7 Oct 2026

Using fine-tuning and min lookahead beam search to improve Whisper

arXiv 2309.10299 ↗First posted 19 Sep 2023eess.AS cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The performance of Whisper in low-resource languages is still far from perfect. In addition to a lack of training data on low-resource languages, we identify some limitations in the beam search algorithm used in Whisper. To address these issues, we fine-tune Whisper on additional data and propose an improved decoding algorithm. On the Vietnamese language, fine-tuning Whisper-Tiny with LoRA leads to an improvement of 38.49 in WER over the zero-shot Whisper-Tiny setting which is a further reduction of 1.45 compared to full-parameter fine-tuning. Additionally, by using Filter-Ends and Min Lookahead decoding algorithms, the WER reduces by 2.26 on average over a range of languages compared to standard beam search. These results generalise to larger Whisper model sizes. We also prove a theorem that Min Lookahead outperforms the standard beam search algorithm used in Whisper.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

12 of 17 references.

References · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  7. 2016Using the Output Embedding to Improve Language Modelscited by 266
  8. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  9. 2023Adaptation of Whisper models to child speech recognitioncited by 25
  10. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  11. 2023A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Modelcited by 19
  12. 2019Speculative Beam Search for Simultaneous Translationcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Fine-tuning Whisper on Low-Resource Languages for Real-World Applicationscited by 11
  2. 2025Advancing STT for Low-Resource Real-World Speechcited by 1
  3. 2025Which one Performs Better? Wav2Vec or Whisper? Applying both in Badini Kurdish Speech to Text (BKSTT)cited by 0