mlsanityData to 7 Oct 2026

Zipformer: A faster and better encoder for automatic speech recognition

arXiv 2310.11230 ↗First posted 17 Oct 2023eess.AS cs.LG cs.SDcited by 77

Not on the map. Why?

Abstract (arXiv)

The Conformer has become the most popular encoder model for automatic speech recognition (ASR). It adds convolution modules to a transformer to learn both local and global dependencies. In this work we describe a faster, more memory-efficient, and better-performing transformer, called Zipformer. Modeling changes include: 1) a U-Net-like encoder structure where middle stacks operate at lower frame rates; 2) reorganized block structure with more modules, within which we re-use attention weights for efficiency; 3) a modified form of LayerNorm called BiasNorm allows us to retain some length information; 4) new activation functions SwooshR and SwooshL work better than Swish. We also propose a new optimizer, called ScaledAdam, which scales the update by each tensor's current scale to keep the relative change about the same, and also explictly learns the parameter scale. It achieves faster convergence and better performance than Adam. Extensive experiments on LibriSpeech, Aishell-1, and WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer over other state-of-the-art ASR models. Our code is publicly available at https://github.com/k2-fsa/icefall.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

34 of 47 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2016Layer Normalizationcited by 5,579
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  11. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  12. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
22 more
  1. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  2. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  3. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  4. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  5. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  6. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  7. 2019QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutionscited by 93
  8. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  9. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  10. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
  11. 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 75
  12. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  13. 2022WeNet 2.0: More Productive End-to-End Speech Recognition Toolkitcited by 46
  14. 2022Pruned RNN-T for fast, memory-efficient ASR trainingcited by 46
  15. 2020Alignment Restricted Streaming Recurrent Neural Network Transducercited by 42
  16. 2021Lhotse: a speech data representation library for the modern deep learning ecosystemcited by 33
  17. 2017Towards End-to-End Speech Recognition with Deep Convolutional Neural Networkscited by 24
  18. 2022Accelerating RNN-T Training and Inference Using CTC guidancecited by 17
  19. 2022Fast and parallel decoding for transducercited by 12
  20. 2022Structured State Space Decoder for Speech Recognition and Synthesiscited by 11
  21. 20223M: Multi-loss, Multi-path and Multi-level Neural Networks for speech recognitioncited by 5
  22. 2022Nextformer: A ConvNeXt Augmented Conformer For End-To-End Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 32; 2026: 17.

77 papers.

Sort:

By area:

  1. 2024GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinementcited by 34
  2. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  3. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  4. 2024Exploring the Capability of Mamba in Speech Applicationscited by 19
  5. 2024Comparing Discrete and Continuous Space LLMs for Speech Recognitioncited by 15
  6. 2024CR-CTC: Consistency regularization on CTC for improved speech recognitioncited by 15
  7. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Modelscited by 14
  8. 2024Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbationcited by 11
  9. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
  10. 2024Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLMcited by 9
  11. 2025From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Modelscited by 9
  12. 2025MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modellingcited by 8
38 more
  1. 2024Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Modelscited by 7
  2. 2024Synchronous Multi-modal Semantic Communication System with Packet-level Codingcited by 7
  3. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matchingcited by 7
  4. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 7
  5. 2024Exploring SSL Discrete Tokens for Multilingual ASRcited by 6
  6. 2025M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesiscited by 6
  7. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  8. 2026Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generationcited by 6
  9. 2024Revisiting ASR Error Correction with Specialized Modelscited by 5
  10. 2025Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysiscited by 5
  11. 2025SongPrep: A Preprocessing Framework and End-to-end Model for Full-song Structure Parsing and Lyrics Transcriptioncited by 5
  12. 2026An Empirical Recipe for Universal Phone Recognitioncited by 5
  13. 2024On the Effectiveness of Acoustic BPE in Decoder-Only TTScited by 4
  14. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 4
  15. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  16. 2025Flow2GAN: Hybrid Flow Matching and GAN with Multi-Resolution Network for Few-step High-Fidelity Audio Generationcited by 4
  17. 2026IndexTTS 2.5 Technical Reportcited by 4
  18. 2026Raon-Speech Technical Reportcited by 4
  19. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 3
  20. 2025VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretrainingcited by 3
  21. 2025Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filteringcited by 3
  22. 2025Revisiting Audio-language Pretraining for Learning General-purpose Audio Representationcited by 3
  23. 2024Multi-Convformer: Extending Conformer with Multiple Convolution Kernelscited by 2
  24. 2024A Language Modeling Approach to Diacritic-Free Hebrew TTScited by 2
  25. 2024Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASRcited by 2
  26. 2024HAINAN: Fast and Accurate Transducer for Hybrid-Autoregressive ASRcited by 2
  27. 2025Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filteringcited by 2
  28. 2025AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuningcited by 2
  29. 2023End-to-end Joint Punctuated and Normalized ASR with a Limited Amount of Punctuated Training Datacited by 1
  30. 2024Self-consistent context aware conformer transducer for speech recognitioncited by 1
  31. 2024The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challengecited by 1
  32. 2024HydraFormer: One Encoder For All Subsampling Ratescited by 1
  33. 2024Linear Time Complexity Conformers with SummaryMixing for Streaming Speech Recognitioncited by 1
  34. 2024Unifying Global and Near-Context Biasing in a Single Trie Passcited by 1
  35. 2024Transducer Consistency Regularization for Speech to Text Applicationscited by 1
  36. 2025Dynamic Acoustic Model Architecture Optimization in Training for ASRcited by 1
  37. 2025Splitformer: An improved early-exit architecture for automatic speech recognition on edge devicescited by 1
  38. 2025Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processingcited by 1

50 most cited of 77 All 77 (JSON)