mlsanityData to 14 Oct 2026

I3D: Transformer architectures with input-dependent dynamic depth for speech recognition

arXiv 2303.07624 ↗First posted 14 Mar 2023cs.CL cs.SD eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

Transformer-based end-to-end speech recognition has achieved great success. However, the large footprint and computational overhead make it difficult to deploy these models in some real-world applications. Model compression techniques can reduce the model size and speed up inference, but the compressed model has a fixed architecture which might be suboptimal. We propose a novel Transformer encoder with Input-Dependent Dynamic Depth (I3D) to achieve strong performance-efficiency trade-offs. With a similar number of layers at inference time, I3D-based models outperform the vanilla Transformer and the static pruned model via iterative layer pruning. We also present interesting analysis on the gate probabilities and the input-dependency, which helps us better understand deep encoders.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 34 of its 40 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

34 of 40 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  9. 2016Deep Networks with Stochastic Depthcited by 979
  10. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  11. 2015Listen, Attend and Spellcited by 543
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
22 more
  1. 2021Dynamic Neural Networks: A Surveycited by 408
  2. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  3. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  4. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 189
  5. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
  6. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  7. 2017Convolutional Networks with Adaptive Inference Graphscited by 148
  8. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  9. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
  10. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  11. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  12. 2021Dynamic Slimmable Networkcited by 83
  13. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  14. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  15. 2020Fractional Skipping: Towards Finer-Grained Dynamic CNN Inferencecited by 23
  16. 2020RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognitioncited by 19
  17. 2021Layer Pruning on Demand with Intermediate CTCcited by 13
  18. 2021Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latencycited by 9
  19. 2021Amortized Neural Networks for Low-Latency Speech Recognitioncited by 9
  20. 2021Bifocal Neural ASR: Exploiting Keyword Spotting for Inference Optimizationcited by 7
  21. 2021Dual-Encoder Architecture with Encoder Selection for Joint Close-Talk and Far-Talk Speech Recognitioncited by 4
  22. 2022Compute Cost Amortized Transformer for Streaming ASRcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 4; 2026: 1.

10 papers.

  1. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  2. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  3. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 37
  4. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  5. 2025Efficient Speech Translation through Model Compression and Knowledge Distillationcited by 5
  6. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  7. 2023Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encodercited by 3
  8. 2025Input Conditioned Layer Dropping in Speech Foundation Modelscited by 2
  9. 2025Intelligent Framework for Human-Robot Collaboration: Dynamic Ergonomics and Adaptive Decision-Makingcited by 0
  10. 2026Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networkscited by 0