# Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST

arXiv: https://arxiv.org/abs/2509.14128 · HTML: https://mlsanity.com/paper/2509.14128/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 26 of 26)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation](https://mlsanity.com/paper/2310.09424/) (2023)
- [Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models](https://mlsanity.com/paper/2407.07263/) (2024)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Lhotse: a speech data representation library for the modern deep learning ecosystem](https://mlsanity.com/paper/2110.12561/) (2021)
- [RNN-T For Latency Controlled ASR With Improved Beam Search](https://mlsanity.com/paper/1911.01629/) (2019)
- [BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5](https://mlsanity.com/paper/2406.19954/) (2024)
- [MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages](https://mlsanity.com/paper/2410.01036/) (2024)
- [One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers](https://mlsanity.com/paper/2506.10766/) (2025)
- [Unified model for code-switching speech recognition and language identification based on a concatenated tokenizer](https://mlsanity.com/paper/2306.08753/) (2023)
- [Word Level Timestamp Generation for Automatic Speech Recognition and Translation](https://mlsanity.com/paper/2505.15646/) (2025)

## Built on by (most cited first, 30 of 60)

- [Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence](https://mlsanity.com/paper/2604.24954/) (2026)
- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation](https://mlsanity.com/paper/2605.19833/) (2026)
- [Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems](https://mlsanity.com/paper/2512.17648/) (2025)
- [Audio Interaction Model](https://mlsanity.com/paper/2606.05121/) (2026)
- [Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models](https://mlsanity.com/paper/2606.14647/) (2026)
- [ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models](https://mlsanity.com/paper/2604.10065/) (2026)
- [Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing](https://mlsanity.com/paper/2607.18934/) (2026)
- [Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception](https://mlsanity.com/paper/2601.09413/) (2026)
- [Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild](https://mlsanity.com/paper/2604.07354/) (2026)
- [GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models](https://mlsanity.com/paper/2606.08194/) (2026)
- [LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization](https://mlsanity.com/paper/2510.23320/) (2025)
- [Efficient Dialect-Aware Modeling and Conditioning for Low-Resource Taiwanese Hakka Speech Processing](https://mlsanity.com/paper/2602.22522/) (2026)
- [Symphony for Speech-to-Text: Supporting Real-Time Medical Voice Interfaces](https://mlsanity.com/paper/2605.16545/) (2026)
- [Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming](https://mlsanity.com/paper/2606.10675/) (2026)
- [Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming](https://mlsanity.com/paper/2606.25460/) (2026)
- [Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs](https://mlsanity.com/paper/2606.29534/) (2026)
- [Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models](https://mlsanity.com/paper/2609.08390/) (2026)
- [Zero-Shot Context-Aware ASR for Diverse Arabic Varieties](https://mlsanity.com/paper/2511.18774/) (2025)
- [From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs](https://mlsanity.com/paper/2512.19161/) (2025)
- [Tagarela - A Portuguese speech dataset from podcasts](https://mlsanity.com/paper/2603.15326/) (2026)
- [SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2603.16924/) (2026)
- [Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models](https://mlsanity.com/paper/2603.25750/) (2026)
- [AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASR](https://mlsanity.com/paper/2604.27543/) (2026)
- [SF20K Competition 2025: Summary and findings](https://mlsanity.com/paper/2605.01496/) (2026)
- [Encoder-Decoder Transformers: Logical Characterizations and Periodicity](https://mlsanity.com/paper/2605.07705/) (2026)
- [Sometin Beta Pass Notin: Improving Multilingual ASR for Nigerian Languages via Knowledge Distillation](https://mlsanity.com/paper/2605.17710/) (2026)
- [Transcribing Children's Speech: ASR Performance and Obtaining Reliable Orthographic Transcriptions](https://mlsanity.com/paper/2605.28833/) (2026)
- [A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026](https://mlsanity.com/paper/2606.03948/) (2026)
- [CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding](https://mlsanity.com/paper/2606.04418/) (2026)

## Abstract (arXiv)

This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encoder and Transformer decoder, it supports 25 languages primarily European. The model was trained on 1.7M hours of total data samples, including Granary and NeMo ASR Set 3.0, with non-speech audio added to reduce hallucinations for ASR and AST. We describe its two-stage pre-training and fine-tuning process with dynamic data balancing, as well as experiments with an nGPT encoder. Results show nGPT scales well with massive data, while FastConformer excels after fine-tuning. For timestamps, Canary-1B-v2 uses the NeMo Forced Aligner (NFA) with an auxiliary CTC model, providing reliable segment-level timestamps for ASR and AST. Evaluations show Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10x faster, and delivers competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large and LLM-based systems. We also release Parakeet-TDT-0.6B-v3, a successor to v2, offering multilingual ASR across the same 25 languages with just 600M parameters.