# Diversity Networks: Neural Network Compression Using Determinantal Point Processes

arXiv: https://arxiv.org/abs/1511.05077 · HTML: https://mlsanity.com/paper/1511.05077/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2015.

## Built on by (most cited first, 25 of 25)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training](https://mlsanity.com/paper/2202.02643/) (2022)
- [Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization](https://mlsanity.com/paper/2105.05612/) (2021)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [Data-Efficient Structured Pruning via Submodular Optimization](https://mlsanity.com/paper/2203.04940/) (2022)
- [Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions](https://mlsanity.com/paper/2209.08554/) (2022)
- [Scaling Up Exact Neural Network Compression by ReLU Stability](https://mlsanity.com/paper/2102.07804/) (2021)
- [Dynamic Sparsity Is Channel-Level Sparsity Learner](https://mlsanity.com/paper/2305.19454/) (2023)
- [Geometry-aware training of factorized layers in tensor Tucker format](https://mlsanity.com/paper/2305.19059/) (2023)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks](https://mlsanity.com/paper/2203.14328/) (2022)
- [Lyapunov Exponents for Diversity in Differentiable Games](https://mlsanity.com/paper/2112.14570/) (2021)
- [Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance](https://mlsanity.com/paper/2203.02770/) (2022)
- [Baseline Pruning-Based Approach to Trojan Detection in Neural Networks](https://mlsanity.com/paper/2101.12016/) (2021)
- [LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification](https://mlsanity.com/paper/2101.02268/) (2021)
- [Task-Oriented Channel Attention for Fine-Grained Few-Shot Classification](https://mlsanity.com/paper/2308.00093/) (2023)
- [Pruning Before Training May Improve Generalization, Provably](https://mlsanity.com/paper/2301.00335/) (2023)
- [Scalable Sampling for Nonsymmetric Determinantal Point Processes](https://mlsanity.com/paper/2201.08417/) (2022)
- [Fragility, Robustness and Antifragility in Deep Learning](https://mlsanity.com/paper/2312.09821/) (2023)
- [Neuronal diversity can improve machine learning for physics and beyond](https://mlsanity.com/paper/2204.04348/) (2022)
- [Diverse Imagenet Models Transfer Better](https://mlsanity.com/paper/2204.09134/) (2022)
- [A pruning method based on the dissimilarity of angle among channels and filters](https://mlsanity.com/paper/2210.16504/) (2022)
- [Pruning Very Deep Neural Network Channels for Efficient Inference](https://mlsanity.com/paper/2211.08339/) (2022)
- [Determinantal Point Process Attention Over Grid Cell Code Supports Out of Distribution Generalization](https://mlsanity.com/paper/2305.18417/) (2023)
- [Neural Network Reduction with Guided Regularizers](https://mlsanity.com/paper/2305.18448/) (2023)

## Abstract (arXiv)

We introduce Divnet, a flexible technique for learning networks with diverse neurons. Divnet models neuronal diversity by placing a Determinantal Point Process (DPP) over neurons in a given layer. It uses this DPP to select a subset of diverse neurons and subsequently fuses the redundant neurons into the selected ones. Compared with previous approaches, Divnet offers a more principled, flexible technique for capturing neuronal diversity and thus implicitly enforcing regularization. This enables effective auto-tuning of network architecture and leads to smaller network sizes without hurting performance. Moreover, through its focus on diversity and neuron fusing, Divnet remains compatible with other procedures that seek to reduce memory footprints of networks. We present experimental results to corroborate our claims: for pruning neural networks, Divnet is seen to be notably superior to competing approaches.