mlsanityData to 14 Oct 2026

Input Conditioned Layer Dropping in Speech Foundation Models

arXiv 2507.07954 ↗First posted 10 Jul 2025cs.SD cs.CV eess.AScited by 2

Not on the map. Why?

Abstract (arXiv)

Curating foundation speech models for edge and IoT settings, where computational resources vary over time, requires dynamic architectures featuring adaptable reduction strategies. One emerging approach is layer dropping (LD) which skips fraction of the layers of a backbone network during inference to reduce the computational load. This allows transforming static models into dynamic ones. However, existing approaches exhibit limitations either in the mode of selecting layers or by significantly modifying the neural architecture. To this end, we propose input-driven LD that employs the network's input features and a lightweight layer selecting network to determine the optimum combination of processing layers. Extensive experimentation on 4 speech and audio public benchmarks, using two different pre-trained foundation models, demonstrates the effectiveness of our approach, thoroughly outperforming random dropping and producing on-par (or better) results to early exit.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

31 of 39 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Squeeze-and-Excitation Networkscited by 4,789
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2016Deep Networks with Stochastic Depthcited by 979
  8. 2021AST: Audio Spectrogram Transformercited by 643
  9. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  10. 2021Dynamic Neural Networks: A Surveycited by 408
  11. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  12. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
19 more
  1. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  2. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 189
  3. 2017Convolutional Networks with Adaptive Inference Graphscited by 148
  4. 2021Learned Token Pruning for Transformerscited by 139
  5. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  6. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 115
  7. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  8. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  9. 2018You Look Twice: GaterNet for Dynamic Filter Selection in CNNscited by 46
  10. 2019Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inferencecited by 31
  11. 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28
  12. 2020Fully Dynamic Inference with Deep Neural Networkscited by 19
  13. 2022Dynamic Split Computing for Efficient Deep Edge Intelligencecited by 18
  14. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  15. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognitioncited by 13
  16. 2023I3D: Transformer architectures with input-dependent dynamic depth for speech recognitioncited by 10
  17. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  18. 2022Bottleneck Low-rank Transformers for Low-resource Spoken Language Understandingcited by 6
  19. 2023Tensor decomposition for minimization of E2E SLU model toward on-device processingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networkscited by 0
  2. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separation
  2. 2026VALSE: Vertical Adaptive Layer Skipping for Efficient Inference in Large Language Models
  3. 2026Does per-frame early exit pay? A compute-matched study of dynamic depth for on-device speech enhancement
  4. 2021Performance optimizations on deep noise suppression models
  5. 2025Splitformer: An improved early-exit architecture for automatic speech recognition on edge devices
  6. 2026Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
  7. 2024Dynamic layer selection in decoder-only transformers
  8. 2024Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity