# Dynamic Acoustic Model Architecture Optimization in Training for ASR

arXiv: https://arxiv.org/abs/2506.13180 · HTML: https://mlsanity.com/paper/2506.13180/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization](https://mlsanity.com/paper/1902.05967/) (2019)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win](https://mlsanity.com/paper/2010.03533/) (2020)
- [OpenELM: An Efficient Language Model Family with Open Training and Inference Framework](https://mlsanity.com/paper/2404.14619/) (2024)
- [Global Sparse Momentum SGD for Pruning Very Deep Neural Networks](https://mlsanity.com/paper/1909.12778/) (2019)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Squeezeformer: An Efficient Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2206.00888/) (2022)
- [Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition](https://mlsanity.com/paper/2109.01163/) (2021)
- [RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition](https://mlsanity.com/paper/1805.05225/) (2018)
- [On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformers](https://mlsanity.com/paper/2011.04906/) (2020)
- [PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation](https://mlsanity.com/paper/2106.14681/) (2021)
- [The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment](https://mlsanity.com/paper/2004.00960/) (2020)
- [Efficient Training of Neural Transducer for Speech Recognition](https://mlsanity.com/paper/2204.10586/) (2022)
- [Gradient-based Intra-attention Pruning on Pre-trained Language Models](https://mlsanity.com/paper/2212.07634/) (2022)
- [Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition](https://mlsanity.com/paper/2407.18930/) (2024)
- [Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search](https://mlsanity.com/paper/2104.05390/) (2021)
- [Similarity and Content-based Phonetic Self Attention for Speech Recognition](https://mlsanity.com/paper/2203.10252/) (2022)
- [Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression](https://mlsanity.com/paper/2501.18895/) (2025)

## Built on by (most cited first, 1 of 1)

- [Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling](https://mlsanity.com/paper/2604.00489/) (2026)

## Abstract (arXiv)

Architecture design is inherently complex. Existing approaches rely on either handcrafted rules, which demand extensive empirical expertise, or automated methods like neural architecture search, which are computationally intensive. In this paper, we introduce DMAO, an architecture optimization framework that employs a grow-and-drop strategy to automatically reallocate parameters during training. This reallocation shifts resources from less-utilized areas to those parts of the model where they are most beneficial. Notably, DMAO only introduces negligible training overhead at a given model complexity. We evaluate DMAO through experiments with CTC on LibriSpeech, TED-LIUM-v2 and Switchboard datasets. The results show that, using the same amount of training resources, our proposed DMAO consistently improves WER by up to 6% relatively across various architectures, model sizes, and datasets. Furthermore, we analyze the pattern of parameter redistribution and uncover insightful findings.