# Subnetwork-to-go: Elastic Neural Network with Dynamic Training and Customizable Inference

arXiv: https://arxiv.org/abs/2312.03464 · HTML: https://mlsanity.com/paper/2312.03464/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [StableMoE: Stable Routing Strategy for Mixture of Experts](https://mlsanity.com/paper/2204.08396/) (2022)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [Sudo rm -rf: Efficient Networks for Universal Audio Source Separation](https://mlsanity.com/paper/2007.06833/) (2020)
- [SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts](https://mlsanity.com/paper/2105.03036/) (2021)
- [Music Demixing Challenge 2021](https://mlsanity.com/paper/2108.13559/) (2021)
- [An efficient encoder-decoder architecture with top-down attention for speech separation](https://mlsanity.com/paper/2209.15200/) (2022)
- [Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network](https://mlsanity.com/paper/2112.02321/) (2021)
- [End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation](https://mlsanity.com/paper/1910.14104/) (2019)
- [Don't shoot butterfly with rifles: Multi-channel Continuous Speech Separation with Early Exit Transformer](https://mlsanity.com/paper/2010.12180/) (2020)
- [Resource-Efficient Neural Architect](https://mlsanity.com/paper/1806.07912/) (2018)
- [Elbert: Fast Albert with Confidence-Window Based Early Exit](https://mlsanity.com/paper/2107.00175/) (2021)
- [HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition](https://mlsanity.com/paper/2204.06328/) (2022)
- [Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation](https://mlsanity.com/paper/2204.00771/) (2022)
- [On the Use of Deep Mask Estimation Module for Neural Source Separation Systems](https://mlsanity.com/paper/2206.07347/) (2022)
- [Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation](https://mlsanity.com/paper/2106.07843/) (2021)
- [Group Communication with Context Codec for Lightweight Source Separation](https://mlsanity.com/paper/2012.07291/) (2020)
- [Elastic Neural Networks for Classification](https://mlsanity.com/paper/1810.00589/) (2018)
- [Elastic Neural Networks: A Scalable Framework for Embedded Computer Vision](https://mlsanity.com/paper/1807.00453/) (2018)
- [Evolving Multi-Resolution Pooling CNN for Monaural Singing Voice Separation](https://mlsanity.com/paper/2008.00816/) (2020)
- [Mixed Precision DNN Qunatization for Overlapped Speech Separation and Recognition](https://mlsanity.com/paper/2111.14479/) (2021)

## Built on by (most cited first, 9 of 9)

- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [Apollo: Band-sequence Modeling for High-Quality Audio Restoration](https://mlsanity.com/paper/2409.08514/) (2024)
- [Gull: A Generative Multifunctional Audio Codec](https://mlsanity.com/paper/2404.04947/) (2024)
- [EAGLE: An Efficient Global Attention Lesion Segmentation Model for Hepatic Echinococcosis](https://mlsanity.com/paper/2506.20333/) (2025)
- [Hierarchical Fusion of Local and Global Visual Features with Mixture-of-Experts for Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2510.27155/) (2025)
- [Multi-Perspective Subimage CLIP with Keyword Guidance for Remote Sensing Image-Text Retrieval](https://mlsanity.com/paper/2601.18190/) (2026)
- [Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores](https://mlsanity.com/paper/2605.12491/) (2026)
- [Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression](https://mlsanity.com/paper/2608.04569/) (2026)
- [Does per-frame early exit pay? A compute-matched study of dynamic depth for on-device speech enhancement](https://mlsanity.com/paper/2609.29867/) (2026)

## Abstract (arXiv)

Deploying neural networks to different devices or platforms is in general challenging, especially when the model size is large or model complexity is high. Although there exist ways for model pruning or distillation, it is typically required to perform a full round of model training or finetuning procedure in order to obtain a smaller model that satisfies the model size or complexity constraints. Motivated by recent works on dynamic neural networks, we propose a simple way to train a large network and flexibly extract a subnetwork from it given a model size or complexity constraint during inference. We introduce a new way to allow a large model to be trained with dynamic depth and width during the training phase, and after the large model is trained we can select a subnetwork from it with arbitrary depth and width during the inference phase with a relatively better performance compared to training the subnetwork independently from scratch. Experiment results on a music source separation model show that our proposed method can effectively improve the separation performance across different subnetwork sizes and complexities with a single large model, and training the large model takes significantly shorter time than training all the different subnetworks.