# Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis

arXiv: https://arxiv.org/abs/2608.18825 · HTML: https://mlsanity.com/paper/2608.18825/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity](https://mlsanity.com/paper/2401.17072/) (2024)
- [The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models](https://mlsanity.com/paper/2402.07658/) (2024)
- [MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder](https://mlsanity.com/paper/2409.14074/) (2024)
- [PoCaP Corpus: A Multimodal Dataset for Smart Operating Room Speech Assistant using Interventional Radiology Workflow Analysis](https://mlsanity.com/paper/2206.12320/) (2022)
- [Speech foundation models in healthcare: Effect of layer selection on pathological speech feature prediction](https://mlsanity.com/paper/2402.01796/) (2024)

## Abstract (arXiv)

Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong generalisation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN->EN+DE continuation, and direct EN+DE fine-tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper-Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two-stage Whisper-Small trajectory shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves.