# Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch

arXiv: https://arxiv.org/abs/2403.14729 · HTML: https://mlsanity.com/paper/2403.14729/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 72 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Data-Driven Sparse Structure Selection for Deep Neural Networks](https://mlsanity.com/paper/1707.01213/) (2017)
- [Discrimination-aware Channel Pruning for Deep Neural Networks](https://mlsanity.com/paper/1810.11809/) (2018)
- [Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimization](https://mlsanity.com/paper/1308.6594/) (2013)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1909.08174/) (2019)
- [SCOP: Scientific Control for Reliable Neural Network Pruning](https://mlsanity.com/paper/2010.10732/) (2020)
- [Channel Pruning via Automatic Structure Search](https://mlsanity.com/paper/2001.08565/) (2020)
- [Operation-Aware Soft Channel Pruning using Differentiable Masks](https://mlsanity.com/paper/2007.03938/) (2020)
- [Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression](https://mlsanity.com/paper/2003.08935/) (2020)

## Built on by (most cited first, 20 of 20)

- [DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models](https://mlsanity.com/paper/2410.11988/) (2024)
- [HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning](https://mlsanity.com/paper/2409.09085/) (2024)
- [All-in-One Tuning and Structural Pruning for Domain-Specific LLMs](https://mlsanity.com/paper/2412.14426/) (2024)
- [ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation](https://mlsanity.com/paper/2505.21817/) (2025)
- [HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworks](https://mlsanity.com/paper/2512.12544/) (2025)
- [Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importance](https://mlsanity.com/paper/2410.16151/) (2024)
- [Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruning](https://mlsanity.com/paper/2506.12041/) (2025)
- [Decay Pruning Method: Smooth Pruning With a Self-Rectifying Procedure](https://mlsanity.com/paper/2406.03879/) (2024)
- [A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseases](https://mlsanity.com/paper/2412.06212/) (2024)
- [One-Cycle Structured Pruning via Stability-Driven Subnetwork Search](https://mlsanity.com/paper/2501.13439/) (2025)
- [Single-weight Model Editing for Post-hoc Spurious Correlation Neutralization](https://mlsanity.com/paper/2501.14182/) (2025)
- [Layer Pruning with Consensus: A Triple-Win Solution](https://mlsanity.com/paper/2411.14345/) (2024)
- [CoSeP: Complementary Separability Pruning via Class-Separability Clustering](https://mlsanity.com/paper/2505.13225/) (2025)
- [Pruning Everything, Everywhere, All at Once](https://mlsanity.com/paper/2506.04513/) (2025)
- [Safe Screening Rules for Group SLOPE](https://mlsanity.com/paper/2506.09451/) (2025)
- [Catalyst: a Novel Regularizer for Structured Pruning with Auxiliary Extension of Parameter Space](https://mlsanity.com/paper/2507.14170/) (2025)
- [Quantization Range Estimation for Convolutional Neural Networks](https://mlsanity.com/paper/2510.04044/) (2025)
- [CAMP-HiVe: Cyclic Pair Merging based Efficient DNN Pruning with Hessian-Vector Approximation for Resource-Constrained Systems](https://mlsanity.com/paper/2511.06265/) (2025)
- [Image Classification via Random Dilated Convolution with Multi-Branch Feature Extraction and Context Excitation](https://mlsanity.com/paper/2604.25188/) (2026)
- [AdaBFL: Multi-Layer Defensive Adaptive Aggregation for Bzantine-Robust Federated Learning](https://mlsanity.com/paper/2604.27434/) (2026)

## Abstract (arXiv)

Current techniques for deep neural network (DNN) pruning often involve intricate multi-step processes that require domain-specific expertise, making their widespread adoption challenging. To address the limitation, the Only-Train-Once (OTO) and OTOv2 are proposed to eliminate the need for additional fine-tuning steps by directly training and compressing a general DNN from scratch. Nevertheless, the static design of optimizers (in OTO) can lead to convergence issues of local optima. In this paper, we proposed the Auto-Train-Once (ATO), an innovative network pruning algorithm designed to automatically reduce the computational and storage costs of DNNs. During the model training phase, our approach not only trains the target model but also leverages a controller network as an architecture generator to guide the learning of target model weights. Furthermore, we developed a novel stochastic gradient algorithm that enhances the coordination between model training and controller network training, thereby improving pruning performance. We provide a comprehensive convergence analysis as well as extensive experiments, and the results show that our approach achieves state-of-the-art performance across various model architectures (including ResNet18, ResNet34, ResNet50, ResNet56, and MobileNetv2) on standard benchmark datasets (CIFAR-10, CIFAR-100, and ImageNet).