mlsanityData to 14 Oct 2026

Cascaded encoders for unifying streaming and non-streaming ASR

arXiv 2010.14606 ↗First posted 27 Oct 2020eess.AS cs.CL cs.SDcited by 53

Not on the map. Why?

Abstract (arXiv)

End-to-end (E2E) automatic speech recognition (ASR) models, by now, have shown competitive performance on several benchmarks. These models are structured to either operate in streaming or non-streaming mode. This work presents cascaded encoders for building a single E2E ASR model that can operate in both these modes simultaneously. The proposed model consists of streaming and non-streaming encoders. Input features are first processed by the streaming encoder; the non-streaming encoder operates exclusively on the output of the streaming encoder. A single decoder then learns to decode either using the output of the streaming or the non-streaming encoder. Results show that this model achieves similar word error rates (WER) as a standalone streaming model when operating in streaming mode, and obtains 10% -- 27% relative improvement when operating in non-streaming mode. Our results also show that the proposed approach outperforms existing E2E two-pass models, especially on long-form speech.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 16; 2023: 17; 2024: 6; 2025: 4; 2026: 1.

53 papers.

Sort:

By area:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  2. 2023End-to-End Speech Recognition: A Surveycited by 129
  3. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  4. 2021Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognitioncited by 25
  5. 2022Pseudo Label Is Better Than Human Labelcited by 25
  6. 2022Streaming End-to-End Multilingual Speech Recognition with Joint Language Identificationcited by 24
  7. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  8. 2022Modular Hybrid Autoregressive Transducercited by 17
  9. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
  10. 2022Variable Attention Masking for Configurable Transformer Transducer Speech Recognitioncited by 14
  11. 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognitioncited by 13
  12. 2021Deliberation of Streaming RNN-Transducer by Non-autoregressive Decodingcited by 13
38 more
  1. 2023Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducercited by 12
  2. 2021FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularizationcited by 11
  3. 2022Improving the fusion of acoustic and text representations in RNN-Tcited by 11
  4. 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASRcited by 11
  5. 2022A Unified Cascaded Encoder ASR Model for Dynamic Model Sizescited by 11
  6. 2022Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word Speech Recognitioncited by 10
  7. 2022Improving Deliberation by Text-Only and Semi-Supervised Trainingcited by 10
  8. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  9. 2022Streaming parallel transducer beam search with fast-slow cascaded encoderscited by 8
  10. 2021Multi-mode Transformer Transducer with Stochastic Future Contextcited by 7
  11. 2021Have best of both worlds: two-pass hybrid and E2E cascading framework for speech recognitioncited by 7
  12. 2023Modular Domain Adaptation for Conformer-Based Streaming ASRcited by 7
  13. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  14. 2022Scaling Up Deliberation for Multilingual ASRcited by 6
  15. 2022Learning a Dual-Mode Speech Recognition Model via Self-Pruningcited by 5
  16. 2022Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systemscited by 5
  17. 2023Text-only Domain Adaptation using Unified Speech-Text Representation in Transducercited by 5
  18. 2024Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASRcited by 5
  19. 2021Dual-Encoder Architecture with Encoder Selection for Joint Close-Talk and Far-Talk Speech Recognitioncited by 4
  20. 2023TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Modelscited by 4
  21. 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
  22. 2021Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarioscited by 3
  23. 2022E2E Segmentation in a Two-Pass Cascaded Encoder ASR Modelcited by 3
  24. 2023UML: A Universal Monolingual Output Layer for Multilingual ASRcited by 3
  25. 2023On Robustness to Missing Video for Audiovisual Speech Recognitioncited by 3
  26. 2023Improved Long-Form Speech Recognition by Jointly Modeling the Primary and Non-primary Speakerscited by 2
  27. 2024Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systemscited by 2
  28. 2023Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASRcited by 1
  29. 2023A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scalecited by 1
  30. 2023Text Injection for Capitalization and Turn-Taking Prediction in Speech Modelscited by 1
  31. 2023On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognitioncited by 1
  32. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  33. 2023Dual Learning for Large Vocabulary On-Device ASRcited by 0
  34. 2023Enhancing the Unified Streaming and Non-streaming Model with Contrastive Learningcited by 0
  35. 2023Partial Rewriting for Multi-Stage ASRcited by 0
  36. 2024Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillationcited by 0
  37. 2024Towards Maximum Likelihood Training for Transducer-based Streaming Speech Recognitioncited by 0
  38. 2025Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Modelscited by 0

50 most cited of 53 All 53 (JSON)