# EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks

arXiv: https://arxiv.org/abs/2006.04270 · HTML: https://mlsanity.com/paper/2006.04270/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2020.

## Built on by (most cited first, 20 of 20)

- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN](https://mlsanity.com/paper/2107.14444/) (2021)
- [Deep Model Compression based on the Training History](https://mlsanity.com/paper/2102.00160/) (2021)
- [S-Rocket: Selective Random Convolution Kernels for Time Series Classification](https://mlsanity.com/paper/2203.03445/) (2022)
- [Perturbation of Deep Autoencoder Weights for Model Compression and Classification of Tabular Data](https://mlsanity.com/paper/2205.08358/) (2022)
- [Not All Attention Is All You Need](https://mlsanity.com/paper/2104.04692/) (2021)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [A Hybrid of Generative and Discriminative Models Based on the Gaussian-coupled Softmax Layer](https://mlsanity.com/paper/2305.05912/) (2023)
- [Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures](https://mlsanity.com/paper/2204.04977/) (2022)
- [Compact Model Training by Low-Rank Projection with Energy Transfer](https://mlsanity.com/paper/2204.05566/) (2022)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach](https://mlsanity.com/paper/2111.02399/) (2021)
- [Latency Adjustable Transformer Encoder for Language Understanding](https://mlsanity.com/paper/2201.03327/) (2022)
- [CALICO: Confident Active Learning with Integrated Calibration](https://mlsanity.com/paper/2407.02335/) (2024)
- [Magnificent Minified Models](https://mlsanity.com/paper/2306.10177/) (2023)
- [Pseudo-label Learning with Calibrated Confidence Using an Energy-based Model](https://mlsanity.com/paper/2404.09585/) (2024)
- [A Comprehensive Study of Structural Pruning for Vision Models](https://mlsanity.com/paper/2406.12315/) (2024)
- [Toward Efficient Convolutional Neural Networks With Structured Ternary Patterns](https://mlsanity.com/paper/2407.14831/) (2024)
- [Playing the Lottery With Concave Regularizers for Sparse Trainable Neural Networks](https://mlsanity.com/paper/2501.11135/) (2025)
- [Fast Tensorization of Neural Networks via Slice-wise Feature Distillation](https://mlsanity.com/paper/2605.19842/) (2026)

## Abstract (arXiv)

Dropout is a well-known regularization method by sampling a sub-network from a larger deep neural network and training different sub-networks on different subsets of the data. Inspired by the dropout concept, we propose EDropout as an energy-based framework for pruning neural networks in classification tasks. In this approach, a set of binary pruning state vectors (population) represents a set of corresponding sub-networks from an arbitrary provided original neural network. An energy loss function assigns a scalar energy loss value to each pruning state. The energy-based model stochastically evolves the population to find states with lower energy loss. The best pruning state is then selected and applied to the original network. Similar to dropout, the kept weights are updated using backpropagation in a probabilistic model. The energy-based model again searches for better pruning states and the cycle continuous. Indeed, this procedure is in fact switching between the energy model, which manages the pruning states, and the probabilistic model, which updates the temporarily unpruned weights, in each iteration. The population can dynamically converge to a pruning state. This can be interpreted as dropout leading to pruning the network. From an implementation perspective, EDropout can prune typical neural networks without modification of the network architecture. We evaluated the proposed method on different flavours of ResNets, AlexNet, and SqueezeNet on the Kuzushiji, Fashion, CIFAR-10, CIFAR-100, and Flowers datasets, and compared the pruning rate and classification performance of the models. On average the networks trained with EDropout achieved a pruning rate of more than 50% of the trainable parameters with approximately <5% and <1% drop of Top-1 and Top-5 classification accuracy, respectively.