# MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder

arXiv: https://arxiv.org/abs/2409.14074 · HTML: https://mlsanity.com/paper/2409.14074/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 96 references that are in the map.
- First posted on arXiv 21 Sep 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation](https://mlsanity.com/paper/1905.03072/) (2019)
- [ASR Error Correction and Domain Adaptation Using Machine Translation](https://mlsanity.com/paper/2003.07692/) (2020)
- [Improved training of end-to-end attention models for speech recognition](https://mlsanity.com/paper/1805.03294/) (2018)
- [AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR](https://mlsanity.com/paper/2310.00274/) (2023)
- [Block Transformer: Global-to-Local Language Modeling for Fast Inference](https://mlsanity.com/paper/2406.02657/) (2024)
- [RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition](https://mlsanity.com/paper/1805.05225/) (2018)
- [Controlling Computation versus Quality for Neural Sequence Models](https://mlsanity.com/paper/2002.07106/) (2020)
- [Speech recognition for medical conversations](https://mlsanity.com/paper/1711.07274/) (2017)
- [RETURNN: The RWTH Extensible Training framework for Universal Recurrent Neural Networks](https://mlsanity.com/paper/1608.00895/) (2016)
- [Robust Prediction of Punctuation and Truecasing for Medical ASR](https://mlsanity.com/paper/2007.02025/) (2020)
- [VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain](https://mlsanity.com/paper/2404.05659/) (2024)
- [Comparing Human and Machine Errors in Conversational Speech Transcription](https://mlsanity.com/paper/1708.08615/) (2017)
- [Performant ASR Models for Medical Entities in Accented Speech](https://mlsanity.com/paper/2406.12387/) (2024)
- [OC16-CE80: A Chinese-English Mixlingual Database and A Speech Recognition Baseline](https://mlsanity.com/paper/1609.08412/) (2016)
- [Medication Regimen Extraction From Medical Conversations](https://mlsanity.com/paper/1912.04961/) (2019)
- [Efficient Utilization of Large Pre-Trained Models for Low Resource ASR](https://mlsanity.com/paper/2210.15445/) (2022)
- [Development of Hybrid ASR Systems for Low Resource Medical Domain Conversational Telephone Speech](https://mlsanity.com/paper/2210.13397/) (2022)

## Built on by (most cited first, 7 of 7)

- [TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Tree](https://mlsanity.com/paper/2508.07014/) (2025)
- [ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark](https://mlsanity.com/paper/2602.12911/) (2026)
- [MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio](https://mlsanity.com/paper/2605.00969/) (2026)
- [FlexCTC: GPU-powered CTC Beam Decoding With Advanced Contextual Abilities](https://mlsanity.com/paper/2508.07315/) (2025)
- [Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks](https://mlsanity.com/paper/2603.22590/) (2026)
- [A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks](https://mlsanity.com/paper/2605.26747/) (2026)
- [Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis](https://mlsanity.com/paper/2608.18825/) (2026)

## Abstract (arXiv)

Multilingual automatic speech recognition (ASR) in the medical domain serves as a foundational task for various downstream applications such as speech translation, spoken language understanding, and voice-activated assistants. This technology improves patient care by enabling efficient communication across language barriers, alleviating specialized workforce shortages, and facilitating improved diagnosis and treatment, particularly during pandemics. In this work, we introduce MultiMed, the first multilingual medical ASR dataset, along with the first collection of small-to-large end-to-end medical ASR models, spanning five languages: Vietnamese, English, German, French, and Mandarin Chinese. To our best knowledge, MultiMed stands as the world's largest medical ASR dataset across all major benchmarks: total duration, number of recording conditions, number of accents, and number of speaking roles. Furthermore, we present the first multilinguality study for medical ASR, which includes reproducible empirical baselines, a monolinguality-multilinguality analysis, Attention Encoder Decoder (AED) vs Hybrid comparative study and a linguistic analysis. We present practical ASR end-to-end training schemes optimized for a fixed number of trainable parameters that are common in industry settings. All code, data, and models are available online: https://github.com/leduckhai/MultiMed/tree/master/MultiMed.