# Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)

arXiv: https://arxiv.org/abs/2606.07608 · HTML: https://mlsanity.com/paper/2606.07608/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 16 of 16)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR](https://mlsanity.com/paper/2406.06619/) (2024)
- [SDS-200: A Swiss German Speech to Standard German Text Corpus](https://mlsanity.com/paper/2205.09501/) (2022)
- [STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions](https://mlsanity.com/paper/2305.18855/) (2023)
- [Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down](https://mlsanity.com/paper/2505.12969/) (2025)
- [Fine-tuning Whisper on Low-Resource Languages for Real-World Applications](https://mlsanity.com/paper/2412.15726/) (2024)
- [H_eval: A new hybrid evaluation metric for automatic speech recognition tasks](https://mlsanity.com/paper/2211.01722/) (2022)
- [Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition](https://mlsanity.com/paper/2505.22251/) (2025)
- [Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning](https://mlsanity.com/paper/2602.06204/) (2026)
- [Spaiche: Extending State-of-the-Art ASR Models to Swiss German Dialects](https://mlsanity.com/paper/2304.11075/) (2023)
- [Does Whisper understand Swiss German? An automatic, qualitative, and human evaluation](https://mlsanity.com/paper/2404.19310/) (2024)
- [Swiss German Speech to Text system evaluation](https://mlsanity.com/paper/2207.00412/) (2022)
- [Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects](https://mlsanity.com/paper/2510.07890/) (2025)
- [Advancing STT for Low-Resource Real-World Speech](https://mlsanity.com/paper/2506.08836/) (2025)

## Built on by (most cited first, 1 of 1)

- [Towards Quantifying Benchmark Optimization in ASR Models](https://mlsanity.com/paper/2608.19936/) (2026)

## Abstract (arXiv)

We present a systematic study of fine-tuning OpenAI's Whisper large-v3 for Swiss German ASR, using 1,367 hours of broadcast speech paired with Standard German subtitles as weak supervision. Through 16 iterative training runs on an NVIDIA DGX Spark (Grace Blackwell, 128 GB unified memory, up to 1 PFLOP FP4), we compare LoRA and full fine-tuning of the 1.55B-parameter model, investigate hallucination root causes, and quantify the effect of data quality, subtitle alignment, and training strategy. Our best model achieves 25.6% measured WER on the All Swiss German Dialects Test Set (ASGDTS) in an honest evaluation on strictly disjoint data. A harmonized error analysis separating genuine errors from valid stylistic variation (tense, word order, Swiss orthography) yields a content WER (cWER) of 13.8%, counting only actual recognition failures. Bias-corrected estimation reduces this to 8.5%, suggesting the true error rate is roughly one third of measured WER. We demonstrate that published state-of-the-art Swiss German ASR results (17.1-17.5% WER) are inflated by benchmark contamination: a vanilla Whisper model self-trained on the ASGDTS test set with zero Swiss German data achieves 13.88% WER, surpassing all published systems. Experiments with Phi-4-multimodal show an even stronger memorization effect (3.9% WER), revealing that the benchmark primarily measures convention matching rather than dialectal comprehension. We release two models, a LoRA adapter (25.32% WER, 13.9% cWER) and a full fine-tuned model (25.60% WER, 13.8% cWER), among the few publicly available, honestly evaluated Whisper models for Swiss German, under Apache 2.0 with full reproducibility, requiring no institutional data agreements.