# Neuron Campaign for Initialization Guided by Information Bottleneck Theory

arXiv: https://arxiv.org/abs/2108.06530 · HTML: https://mlsanity.com/paper/2108.06530/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 14 Aug 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Graph Information Bottleneck](https://mlsanity.com/paper/2010.12811/) (2020)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)
- [On the Impact of the Activation Function on Deep Neural Networks Training](https://mlsanity.com/paper/1902.06853/) (2019)
- [TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Data](https://mlsanity.com/paper/2106.03096/) (2021)
- [Data-dependent Initializations of Convolutional Neural Networks](https://mlsanity.com/paper/1511.06856/) (2015)
- [TSSRGCN: Temporal Spectral Spatial Retrieval Graph Convolutional Network for Traffic Flow Forecasting](https://mlsanity.com/paper/2011.14638/) (2020)
- [Tag2Vec: Learning Tag Representations in Tag Networks](https://mlsanity.com/paper/1905.03041/) (2019)

## Built on by (most cited first, 9 of 9)

- [Brain Network Transformer](https://mlsanity.com/paper/2210.06681/) (2022)
- [Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective](https://mlsanity.com/paper/2203.03871/) (2022)
- [Company-as-Tribe: Company Financial Risk Assessment on Tribe-Style Graph with Hierarchical Graph Neural Networks](https://mlsanity.com/paper/2301.13492/) (2023)
- [Predicting the Silent Majority on Graphs: Knowledge Transferable Graph Neural Network](https://mlsanity.com/paper/2302.00873/) (2023)
- [Gaussian Pre-Activations in Neural Networks: Myth or Reality?](https://mlsanity.com/paper/2205.12379/) (2022)
- [GraphAD: A Graph Neural Network for Entity-Wise Multivariate Time-Series Anomaly Detection](https://mlsanity.com/paper/2205.11139/) (2022)
- [Learning Rate Perturbation: A Generic Plugin of Learning Rate Schedule towards Flatter Local Minima](https://mlsanity.com/paper/2208.11873/) (2022)
- [CyclicFL: A Cyclic Model Pre-Training Approach to Efficient Federated Learning](https://mlsanity.com/paper/2301.12193/) (2023)
- [Justices for Information Bottleneck Theory](https://mlsanity.com/paper/2305.11387/) (2023)

## Abstract (arXiv)

Initialization plays a critical role in the training of deep neural networks (DNN). Existing initialization strategies mainly focus on stabilizing the training process to mitigate gradient vanish/explosion problems. However, these initialization methods are lacking in consideration about how to enhance generalization ability. The Information Bottleneck (IB) theory is a well-known understanding framework to provide an explanation about the generalization of DNN. Guided by the insights provided by IB theory, we design two criteria for better initializing DNN. And we further design a neuron campaign initialization algorithm to efficiently select a good initialization for a neural network on a given dataset. The experiments on MNIST dataset show that our method can lead to a better generalization performance with faster convergence.