mlsanityData to 14 Oct 2026

Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST

arXiv 2509.14128 ↗First posted 17 Sep 2025cs.CL eess.AScited by 60

Not on the map. Why?

Abstract (arXiv)

This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encoder and Transformer decoder, it supports 25 languages primarily European. The model was trained on 1.7M hours of total data samples, including Granary and NeMo ASR Set 3.0, with non-speech audio added to reduce hallucinations for ASR and AST. We describe its two-stage pre-training and fine-tuning process with dynamic data balancing, as well as experiments with an nGPT encoder. Results show nGPT scales well with massive data, while FastConformer excels after fine-tuning. For timestamps, Canary-1B-v2 uses the NeMo Forced Aligner (NFA) with an auxiliary CTC model, providing reliable segment-level timestamps for ASR and AST. Evaluations show Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10x faster, and delivers competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large and LLM-based systems. We also release Parakeet-TDT-0.6B-v3, a successor to v2, offering multilingual ASR across the same 25 languages with just 600M parameters.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

26 of 42 references.

References · 26

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  8. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  11. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
14 more
  1. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  2. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  3. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 124
  4. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  5. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  6. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
  7. 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 58
  8. 2021Lhotse: a speech data representation library for the modern deep learning ecosystemcited by 33
  9. 2019RNN-T For Latency Controlled ASR With Improved Beam Searchcited by 20
  10. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 19
  11. 2024MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languagescited by 14
  12. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  13. 2023Unified model for code-switching speech recognition and language identification based on a concatenated tokenizercited by 8
  14. 2025Word Level Timestamp Generation for Automatic Speech Recognition and Translationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 55.

60 papers.

Sort:

By area:

  1. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 30
  2. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 7
  3. 2026Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulationcited by 5
  4. 2025Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systemscited by 4
  5. 2026Audio Interaction Modelcited by 4
  6. 2026Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Modelscited by 4
  7. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  8. 2026Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timingcited by 3
  9. 2026Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perceptioncited by 2
  10. 2026Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wildcited by 2
  11. 2026GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Modelscited by 2
  12. 2025LibriConvo: Simulating Conversations from Read Literature for ASR and Diarizationcited by 1
38 more
  1. 2026Efficient Dialect-Aware Modeling and Conditioning for Low-Resource Taiwanese Hakka Speech Processingcited by 1
  2. 2026Symphony for Speech-to-Text: Supporting Real-Time Medical Voice Interfacescited by 1
  3. 2026Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programmingcited by 1
  4. 2026Fully Differentiable Neural Forced Alignment via Soft Dynamic Programmingcited by 1
  5. 2026Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMscited by 1
  6. 2026Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Modelscited by 1
  7. 2025Zero-Shot Context-Aware ASR for Diverse Arabic Varietiescited by 0
  8. 2025From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programscited by 0
  9. 2026Tagarela - A Portuguese speech dataset from podcastscited by 0
  10. 2026SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translationcited by 0
  11. 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
  12. 2026AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASRcited by 0
  13. 2026SF20K Competition 2025: Summary and findingscited by 0
  14. 2026Encoder-Decoder Transformers: Logical Characterizations and Periodicitycited by 0
  15. 2026Sometin Beta Pass Notin: Improving Multilingual ASR for Nigerian Languages via Knowledge Distillationcited by 0
  16. 2026Transcribing Children's Speech: ASR Performance and Obtaining Reliable Orthographic Transcriptionscited by 0
  17. 2026A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026cited by 0
  18. 2026CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encodingcited by 0
  19. 2026TRADE: Transducer-Augmented Decoder for Speech LLMcited by 0
  20. 2026ROMPAR: Morphological Completion and Demographic Unlearning for Romanian-Accented Speech Recognitioncited by 0
  21. 2026MLLP-VRAIN UPV system for the IWSLT 2026 Simultaneous Speech Translation taskcited by 0
  22. 2026ASTRA: A Scalable Next-Generation ATCO Training Simulator with Autonomous Simpilotscited by 0
  23. 2026RedVox: Safety and Fairness Gaps in Speech Models Across Languagescited by 0
  24. 2026What Counts as an Error? Dual-Reference Benchmarking for Atypical ASRcited by 0
  25. 2026Noisy-Channel Minimum Bayes Risk Decodingcited by 0
  26. 2026The Role of Disfluencies in Speech Translationcited by 0
  27. 2026Position: AI Leaderboards Are Underserving the Global South: A Case Study from Indiacited by 0
  28. 2026PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understandingcited by 0
  29. 2026Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speechcited by 0
  30. 2026Building a Production Greek-English Speech Recognizercited by 0
  31. 2026RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialoguecited by 0
  32. 2026Beyond EER: Multi-Dimensional Evaluation of Information Leakage in Speaker De-Identificationcited by 0
  33. 2026The Hidden Cost of Digits: Number Normalization and WER in ASR Systemscited by 0
  34. 2026Reusing Latent Speech Representations for Query-Conditioned Topic Localization in Transcriptscited by 0
  35. 2026Controlling Backchannels in Streamable Full-duplex Modelscited by 0
  36. 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speechcited by 0
  37. 2026How to Reduce Whisper Hallucinationcited by 0
  38. 2026CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settingscited by 0

50 most cited of 60 All 60 (JSON)