# Reducing the Model Order of Deep Neural Networks Using Information Theory

arXiv: https://arxiv.org/abs/1605.04859 · HTML: https://mlsanity.com/paper/1605.04859/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2016.

## Built on by (most cited first, 10 of 10)

- [Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning](https://mlsanity.com/paper/2109.05687/) (2021)
- [Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks](https://mlsanity.com/paper/2404.08347/) (2024)
- [FIT: A Metric for Model Sensitivity](https://mlsanity.com/paper/2210.08502/) (2022)
- [LayerNorm: A key component in parameter-efficient fine-tuning](https://mlsanity.com/paper/2403.20284/) (2024)
- [DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS](https://mlsanity.com/paper/2106.11655/) (2021)
- [Weight Pruning and Uncertainty in Radio Galaxy Classification](https://mlsanity.com/paper/2111.11654/) (2021)
- [Much Easier Said Than Done: Falsifying the Causal Relevance of Linear Decoding Methods](https://mlsanity.com/paper/2211.04367/) (2022)
- [X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity](https://mlsanity.com/paper/2310.17166/) (2023)
- [Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds](https://mlsanity.com/paper/2505.13614/) (2025)
- [Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks](https://mlsanity.com/paper/2511.04494/) (2025)

## Abstract (arXiv)

Deep neural networks are typically represented by a much larger number of parameters than shallow models, making them prohibitive for small footprint devices. Recent research shows that there is considerable redundancy in the parameter space of deep neural networks. In this paper, we propose a method to compress deep neural networks by using the Fisher Information metric, which we estimate through a stochastic optimization method that keeps track of second-order information in the network. We first remove unimportant parameters and then use non-uniform fixed point quantization to assign more bits to parameters with higher Fisher Information estimates. We evaluate our method on a classification task with a convolutional neural network trained on the MNIST data set. Experimental results show that our method outperforms existing methods for both network pruning and quantization.