# Whale: Large-Scale multilingual ASR model with w2v-BERT and E-Branchformer with large speech data

arXiv: https://arxiv.org/abs/2506.01439 · HTML: https://mlsanity.com/paper/2506.01439/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage](https://mlsanity.com/paper/2111.09344/) (2021)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [Multilingual Speech Recognition With A Single End-To-End Model](https://mlsanity.com/paper/1711.01694/) (2017)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [mHuBERT-147: A Compact Multilingual HuBERT Model](https://mlsanity.com/paper/2406.06371/) (2024)
- [YODAS: Youtube-Oriented Dataset for Audio and Speech](https://mlsanity.com/paper/2406.00899/) (2024)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data](https://mlsanity.com/paper/2309.13876/) (2023)
- [OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification](https://mlsanity.com/paper/2402.12654/) (2024)
- [JTubeSpeech: corpus of Japanese speech collected from YouTube for speech recognition and speaker verification](https://mlsanity.com/paper/2112.09323/) (2021)
- [The Norwegian Parliamentary Speech Corpus](https://mlsanity.com/paper/2201.10881/) (2022)
- [Better Intermediates Improve CTC Inference](https://mlsanity.com/paper/2204.00176/) (2022)
- [Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting](https://mlsanity.com/paper/2406.12611/) (2024)

## Built on by (most cited first, 3 of 3)

- [Listening, Imagining & Refining: A Heuristic Optimized ASR Correction Framework with LLMs](https://mlsanity.com/paper/2509.15095/) (2025)
- [Index-MSR: A high-efficiency multimodal fusion framework for speech recognition](https://mlsanity.com/paper/2509.22744/) (2025)
- [Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning](https://mlsanity.com/paper/2609.13045/) (2026)

## Abstract (arXiv)

This paper reports on the development of a large-scale speech recognition model, Whale. Similar to models such as Whisper and OWSM, Whale leverages both a large model size and a diverse, extensive dataset. Whale's architecture integrates w2v-BERT self-supervised model, an encoder-decoder backbone built on E-Branchformer, and a joint CTC-attention decoding strategy. The training corpus comprises varied speech data, of not only public corpora but also in-house data, thereby enhancing the model's robustness to different speaking styles and acoustic conditions. Through evaluations on multiple benchmarks, Whale achieved comparable performance to existing models. In particular, it achieves a word error rate of 2.4% on the Librispeech test-clean set and a character error rate of 3.4% on the CSJ eval3 set, outperforming Whisper large-v3 and OWSM v3.1.