mlsanityData to 14 Oct 2026

Neuron Campaign for Initialization Guided by Information Bottleneck Theory

arXiv 2108.06530 ↗First posted 14 Aug 2021cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Initialization plays a critical role in the training of deep neural networks (DNN). Existing initialization strategies mainly focus on stabilizing the training process to mitigate gradient vanish/explosion problems. However, these initialization methods are lacking in consideration about how to enhance generalization ability. The Information Bottleneck (IB) theory is a well-known understanding framework to provide an explanation about the generalization of DNN. Guided by the insights provided by IB theory, we design two criteria for better initializing DNN. And we further design a neuron campaign initialization algorithm to efficiently select a good initialization for a neural network on a given dataset. The experiments on MNIST dataset show that our method can lead to a better generalization performance with faster convergence.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 14 Aug 2021.

Builds on

18 of 27 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  6. 2016An overview of gradient descent optimization algorithmscited by 1,033
  7. 2016Deep Variational Information Bottleneckcited by 930
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2018MINE: Mutual Information Neural Estimationcited by 683
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  11. 2020Graph Information Bottleneckcited by 168
  12. 2015All you need is a good initcited by 124
6 more
  1. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 105
  2. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  3. 2021TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Datacited by 36
  4. 2015Data-dependent Initializations of Convolutional Neural Networkscited by 29
  5. 2020TSSRGCN: Temporal Spectral Spatial Retrieval Graph Convolutional Network for Traffic Flow Forecastingcited by 14
  6. 2019Tag2Vec: Learning Tag Representations in Tag Networkscited by 8

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 4.

9 papers.

  1. 2022Brain Network Transformercited by 94
  2. 2022Discriminability-Transferability Trade-Off: An Information-Theoretic Perspectivecited by 13
  3. 2023Company-as-Tribe: Company Financial Risk Assessment on Tribe-Style Graph with Hierarchical Graph Neural Networkscited by 10
  4. 2023Predicting the Silent Majority on Graphs: Knowledge Transferable Graph Neural Networkcited by 10
  5. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  6. 2022GraphAD: A Graph Neural Network for Entity-Wise Multivariate Time-Series Anomaly Detectioncited by 5
  7. 2022Learning Rate Perturbation: A Generic Plugin of Learning Rate Schedule towards Flatter Local Minimacited by 1
  8. 2023CyclicFL: A Cyclic Model Pre-Training Approach to Efficient Federated Learningcited by 0
  9. 2023Justices for Information Bottleneck Theorycited by 0