# Anatomy of Industrial Scale Multilingual ASR

arXiv: https://arxiv.org/abs/2404.09841 · HTML: https://mlsanity.com/paper/2404.09841/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [Why does CTC result in peaky behavior?](https://mlsanity.com/paper/2105.14849/) (2021)
- [Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learning](https://mlsanity.com/paper/2110.00165/) (2021)
- [ASR2K: Speech Recognition for Around 2000 Languages without Audio](https://mlsanity.com/paper/2209.02842/) (2022)
- [Pseudo Label Is Better Than Human Label](https://mlsanity.com/paper/2203.12668/) (2022)
- [On Addressing Practical Challenges for RNN-Transducer](https://mlsanity.com/paper/2105.00858/) (2021)
- [Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks](https://mlsanity.com/paper/2210.07499/) (2022)
- [Conformer-1: Robust ASR via Large-Scale Semisupervised Bootstrapping](https://mlsanity.com/paper/2404.07341/) (2024)
- [Neural Transducer Training: Reduced Memory Consumption with Sample-wise Computation](https://mlsanity.com/paper/2211.16270/) (2022)

## Built on by (most cited first, 14 of 14)

- [GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement](https://mlsanity.com/paper/2406.11546/) (2024)
- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch](https://mlsanity.com/paper/2412.08237/) (2024)
- [MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyond](https://mlsanity.com/paper/2412.11538/) (2024)
- [VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining](https://mlsanity.com/paper/2505.21527/) (2025)
- [Early Attentive Sparsification Accelerates Neural Speech Transcription](https://mlsanity.com/paper/2506.15912/) (2025)
- [SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge](https://mlsanity.com/paper/2507.03343/) (2025)
- [Efficient ASR for Low-Resource Languages: Leveraging Cross-Lingual Unlabeled Data](https://mlsanity.com/paper/2512.07277/) (2025)
- [Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for Polish](https://mlsanity.com/paper/2408.00005/) (2024)
- [SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2506.12672/) (2025)
- [From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs](https://mlsanity.com/paper/2512.19161/) (2025)
- [Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Models](https://mlsanity.com/paper/2606.11639/) (2026)
- [Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correction](https://mlsanity.com/paper/2608.22196/) (2026)
- [FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speech](https://mlsanity.com/paper/2609.32396/) (2026)

## Abstract (arXiv)

This paper describes AssemblyAI's industrial-scale automatic speech recognition (ASR) system, designed to meet the requirements of large-scale, multilingual ASR serving various application needs. Our system leverages a diverse training dataset comprising unsupervised (12.5M hours), supervised (188k hours), and pseudo-labeled (1.6M hours) data across four languages. We provide a detailed description of our model architecture, consisting of a full-context 600M-parameter Conformer encoder pre-trained with BEST-RQ and an RNN-T decoder fine-tuned jointly with the encoder. Our extensive evaluation demonstrates competitive word error rates (WERs) against larger and more computationally expensive models, such as Whisper large and Canary-1B. Furthermore, our architectural choices yield several key advantages, including an improved code-switching capability, a 5x inference speedup compared to an optimized Whisper baseline, a 30% reduction in hallucination rate on speech data, and a 90% reduction in ambient noise compared to Whisper, along with significantly improved time-stamp accuracy. Throughout this work, we adopt a system-centric approach to analyzing various aspects of fully-fledged ASR models to gain practically relevant insights useful for real-world services operating at scale.