Entropic alternatives to initialization
Not on the map. Why?
Abstract (arXiv)
Local entropic loss functions provide a versatile framework to define architecture-aware regularization procedures. Besides the possibility of being anisotropic in the synaptic space, the local entropic smoothening of the loss function can vary during training, thus yielding a tunable model complexity. A scoping protocol where the regularization is strong in the early-stage of the training and then fades progressively away constitutes an alternative to standard initialization procedures for deep convolutional neural networks, nonetheless, it has wider applicability. We analyze anisotropic, local entropic smoothenings in the language of statistical physics and information theory, providing insight into both their interpretation and workings. We comment some aspects related to the physics of renormalization and the spacetime structure of convolutional networks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 16 Jul 2021.
Builds on
16 of 32 references.
References · 16
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
- 2016Deep Information Propagationcited by 194
- 2013Revisiting Natural Gradient for Deep Networkscited by 157
- 2014Deep learning with Elastic Averaging SGDcited by 90
- 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
- 2016Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemescited by 40
- 2015Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapsescited by 39
- 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22
- 2015Local entropy as a measure for sampling solutions in Constraint Satisfaction Problemscited by 20
- 2020A scale-dependent notion of effective dimensioncited by 17
4 more
- 2019Mean-field inference methods for neural networkscited by 16
- 2021Towards quantifying information flows: relative entropy in deep neural networks and the renormalization groupcited by 12
- 2020Partial local entropy and anisotropy in deep weight spacescited by 3
- 2016Distributed stochastic optimization for deep learning (thesis)cited by 1
Built on by
Not cited yet.