mlsanityData to 7 Oct 2026

A Unified Cascaded Encoder ASR Model for Dynamic Model Sizes

arXiv 2204.06164 ↗First posted 13 Apr 2022eess.AS cs.LG cs.SDcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a dynamic cascaded encoder Automatic Speech Recognition (ASR) model, which unifies models for different deployment scenarios. Moreover, the model can significantly reduce model size and power consumption without loss of quality. Namely, with the dynamic cascaded encoder model, we explore three techniques to maximally boost the performance of each model size: 1) Use separate decoders for each sub-model while sharing the encoders; 2) Use funnel-pooling to improve the encoder efficiency; 3) Balance the size of causal and non-causal encoders to improve quality and fit deployment constraints. Overall, the proposed large-medium model has 30% smaller size and reduces power consumption by 33%, compared to the baseline cascaded encoder model. The triple-size model that unifies the large, medium, and small models achieves 37% total size reduction with minimal quality loss, while substantially reducing the engineering efforts of having separate models.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

21 of 30 references.

References · 21

  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2015Attention-Based Models for Speech Recognitioncited by 400
  4. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  5. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  6. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  7. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  8. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  9. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  10. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  11. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  12. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
9 more
  1. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  2. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 51
  3. 2021Tied & Reduced RNN-T Decodercited by 35
  4. 2020A New Training Pipeline for an Improved Neural Transducercited by 34
  5. 2020Dynamic Sparsity Neural Networks for Automatic Speech Recognitioncited by 23
  6. 2021Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR via Supernetcited by 12
  7. 2021Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latencycited by 9
  8. 2021CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognitioncited by 8
  9. 2021Collaborative Training of Acoustic Encoders for Speech Recognitioncited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 4; 2025: 1.

11 papers.

  1. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  2. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  3. 2024Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Modelscited by 5
  4. 2023TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Modelscited by 4
  5. 2022E2E Segmentation in a Two-Pass Cascaded Encoder ASR Modelcited by 3
  6. 2023Semantic Segmentation with Bidirectional Language Models Improves Long-form ASRcited by 2
  7. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  8. 2023RAND: Robustness Aware Norm Decay For Quantized Seq2seq Modelscited by 0
  9. 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layercited by 0
  10. 2024Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillationcited by 0
  11. 2024Federated Learning of Large ASR Models in the Real Worldcited by 0