mlsanityData to 14 Oct 2026

Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillation

arXiv 2405.13514 ↗First posted 22 May 2024eess.AS cs.CL cs.SDcited by 0

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

End-to-end (E2E) automatic speech recognition (ASR) can operate in two modes: streaming and non-streaming, each with its pros and cons. Streaming ASR processes the speech frames in real-time as it is being received, while non-streaming ASR waits for the entire speech utterance; thus, professionals may have to operate in either mode to satisfy their application. In this work, we present joint optimization of streaming and non-streaming ASR based on multi-decoder and knowledge distillation. Primarily, we study 1) the encoder integration of these ASR modules, followed by 2) separate decoders to make the switching mode flexible, and enhancing performance by 3) incorporating similarity-preserving knowledge distillation between the two modular encoders and decoders. Evaluation results show 2.6%-5.3% relative character error rate reductions (CERR) on CSJ for streaming ASR, and 8.3%-9.7% relative CERRs for non-streaming ASR within a single model compared to multiple standalone modules.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 23 of its 32 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

23 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Conformer with dual-mode chunked attention for joint online and offline ASR22 Jun

Other references · 22

  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  3. 2015Listen, Attend and Spellcited by 543
  4. 2019Similarity-Preserving Knowledge Distillationcited by 516
  5. 2015Attention-Based Models for Speech Recognitioncited by 401
  6. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  7. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
  8. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  9. 2023End-to-End Speech Recognition: A Surveycited by 129
  10. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 96
  11. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  12. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 76
10 more
  1. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  2. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 69
  3. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 69
  4. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 53
  5. 2020Towards Fast and Accurate Streaming End-to-End ASRcited by 47
  6. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 38
  7. 2019Transformer ASR with Contextual Block Processingcited by 28
  8. 2022A Unified Cascaded Encoder ASR Model for Dynamic Model Sizescited by 11
  9. 2022Streaming parallel transducer beam search with fast-slow cascaded encoderscited by 8
  10. 20224D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoderscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer
  2. 2023Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation
  3. 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model
  4. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data
  5. 2023On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition
  6. 2021Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
  7. 2024Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper
  8. 2022Improving Streaming End-to-End ASR on Transformer-based Causal Models with Encoder States Revision Strategies