mlsanityData to 14 Oct 2026

Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition

arXiv 2012.05481 ↗First posted 10 Dec 2020cs.SD cs.CL eess.AScited by 39

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a novel two-pass approach to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model. Our model adopts the hybrid CTC/attention architecture, in which the conformer layers in the encoder are modified. We propose a dynamic chunk-based attention strategy to allow arbitrary right context length. At inference time, the CTC decoder generates n-best hypotheses in a streaming way. The inference latency could be easily controlled by only changing the chunk size. The CTC hypotheses are then rescored by the attention decoder to get the final result. This efficient rescoring process causes very little sentence-level latency. Our experiments on the open 170-hour AISHELL-1 dataset show that, the proposed method can unify the streaming and non-streaming model simply and efficiently. On the AISHELL-1 test set, our unified model achieves 5.60% relative character error rate (CER) reduction in non-streaming ASR compared to a standard non-streaming transformer. The same model achieves 5.42% CER with 640ms latency in a streaming ASR system.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 8; 2023: 11; 2024: 4; 2025: 4; 2026: 4.

39 papers.

Sort:

By area:

  1. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 125
  2. 2021ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversationcited by 44
  3. 2021U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognitioncited by 29
  4. 2022Language-specific Characteristic Assistance for Code-switching Speech Recognitioncited by 20
  5. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  6. 2023ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMscited by 13
  7. 2021WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognitioncited by 10
  8. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  9. 2024Mamba for Streaming ASR Combined with Unimodal Aggregationcited by 8
  10. 2021SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memorycited by 7
  11. 2022Delay-penalized transducer for low-latency streaming ASRcited by 7
  12. 2023Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Studycited by 7
27 more
  1. 2023Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognitioncited by 7
  2. 2025Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correctioncited by 7
  3. 2021Decoupling recognition and transcription in Mandarin ASRcited by 6
  4. 2022Improving Mandarin Speech Recogntion with Block-augmented Transformercited by 6
  5. 2022Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillationcited by 6
  6. 2023DualVC: Dual-mode Voice Conversion using Intra-model Knowledge Distillation and Hybrid Predictive Codingcited by 4
  7. 2023Hyper-parameter Adaptation of Conformer ASR Systems for Elderly and Dysarthric Speech Recognitioncited by 4
  8. 2022E2E Segmentation in a Two-Pass Cascaded Encoder ASR Modelcited by 3
  9. 2022Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASRcited by 2
  10. 2024UCorrect: An Unsupervised Framework for Automatic Speech Recognition Error Correctioncited by 2
  11. 2024Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognitioncited by 2
  12. 2025Chunk Based Speech Pre-training with High Resolution Finite Scalar Quantizationcited by 2
  13. 2022SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolutioncited by 1
  14. 2023Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognitioncited by 1
  15. 2023Perception and Semantic Aware Regularization for Sequential Confidence Calibrationcited by 1
  16. 2024R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translationcited by 1
  17. 2025Unifying Streaming and Non-streaming Zipformer-based ASRcited by 1
  18. 2025In-domain SSL pre-training and streaming ASRcited by 1
  19. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASRcited by 1
  20. 2022Linguistic-Enhanced Transformer with CTC Embedding for Speech Recognitioncited by 0
  21. 2023Enhancing the Unified Streaming and Non-streaming Model with Contrastive Learningcited by 0
  22. 2023DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASRcited by 0
  23. 2023ApproBiVT: Lead ASR Models to Generalize Better Using Approximated Bias-Variance Tradeoff Guided Early Stopping and Checkpoint Averagingcited by 0
  24. 2023Improving CTC-AED model with integrated-CTC and auxiliary loss regularizationcited by 0
  25. 2026SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognitioncited by 0
  26. 2026TRADE: Transducer-Augmented Decoder for Speech LLMcited by 0
  27. 2026O'ido: Open-Vocabulary Speech Recognition on a 5 Microcontroller without a Neural Acceleratorcited by 0