mlsanityData to 14 Oct 2026

Conformer with dual-mode chunked attention for joint online and offline ASR

arXiv 2206.11157 ↗First posted 22 Jun 2022eess.AScited by 10

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

In this paper, we present an in-depth study on online attention mechanisms and distillation techniques for dual-mode (i.e., joint online and offline) ASR using the Conformer Transducer. In the dual-mode Conformer Transducer model, layers can function in online or offline mode while sharing parameters, and in-place knowledge distillation from offline to online mode is applied in training to improve online accuracy. In our study, we first demonstrate accuracy improvements from using chunked attention in the Conformer encoder compared to autoregressive attention with and without lookahead. Furthermore, we explore the efficient KLD and 1-best KLD losses with different shifts between online and offline outputs in the knowledge distillation. Finally, we show that a simplified dual-mode Conformer that only has mode-specific self-attention performs equally well as the one also having mode-specific convolutions and normalization. Our experiments are based on two very different datasets: the Librispeech task and an internal corpus of medical conversations. Results show that the proposed dual-mode system using chunked attention yields 5% and 4% relative WER improvement on the Librispeech and medical tasks, compared to the dual-mode system using autoregressive attention with similar average lookahead.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 20 of its 28 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

20 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models7 Oct

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  4. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  5. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 113
  6. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  7. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 97
  8. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 69
  9. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 69
  10. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 69
  11. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 63
  12. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 44
7 more
  1. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 38
  2. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 28
  3. 2020Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognitioncited by 24
  4. 2020Universal ASR: Unifying Streaming and Non-Streaming ASR Using a Single Encoder-Decoder Modelcited by 9
  5. 2019Listen, Attend, Spell and Adapt: Speaker Adapted Sequence-to-Sequence ASRcited by 8
  6. 2021Multi-mode Transformer Transducer with Stochastic Future Contextcited by 7
  7. 2021Dual-Encoder Architecture with Encoder Selection for Joint Close-Talk and Far-Talk Speech Recognitioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 2; 2025: 1; 2026: 2.

10 papers.

  1. 2023Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognitioncited by 7
  2. 2022Learning a Dual-Mode Speech Recognition Model via Self-Pruningcited by 5
  3. 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
  4. 2023Multi-pass Training and Cross-information Fusion for Low-resource End-to-end Accented Speech Recognitioncited by 3
  5. 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layercited by 2
  6. 2025Chunk Based Speech Pre-training with High Resolution Finite Scalar Quantizationcited by 2
  7. 2022SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolutioncited by 1
  8. 2024Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillationcited by 0
  9. 2026TRADE: Transducer-Augmented Decoder for Speech LLMcited by 0
  10. 2026Alignment-Path Distillation from Non-streaming ASR-LLMs for Streaming Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition
  2. 2023Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation
  3. 2023Augmenting conformers with structured state-space sequence models for online speech recognition
  4. 20224-bit Conformer with Native Quantization Aware Training for Speech Recognition
  5. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data
  6. 2023Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation
  7. 2023On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition
  8. 2019Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hopping