Understanding and Leveraging the Learning Phases of Neural Networks
Not on the map. Why?
Abstract (arXiv)
The learning dynamics of deep neural networks are not well understood. The information bottleneck (IB) theory proclaimed separate fitting and compression phases. But they have since been heavily debated. We comprehensively analyze the learning dynamics by investigating a layer's reconstruction ability of the input and prediction performance based on the evolution of parameters during training. We empirically show the existence of three phases using common datasets and architectures such as ResNet and VGG: (i) near constant reconstruction loss, (ii) decrease, and (iii) increase. We also derive an empirically grounded data model and prove the existence of phases for single-layer networks. Technically, our approach leverages classical complexity analysis. It differs from IB by relying on measuring reconstruction loss rather than information theoretic measures to relate information of intermediate layers and inputs. Our work implies a new best practice for transfer learning: We show empirically that the pre-training of a classifier should stop well before its performance is optimal.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 11 Dec 2023.
Builds on
21 of 31 references.
References · 21
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2015Deep Learning and the Information Bottleneck Principlecited by 802
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
- 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
- 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
- 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
9 more
- 2018Generalization Error in Deep Learningcited by 47
- 2016Optimality of the Johnson-Lindenstrauss Lemmacited by 42
- 2020On Information Plane Analyses of Neural Network Classifiers -- A Reviewcited by 37
- 2021PAC-Bayes Information Bottleneckcited by 28
- 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networkscited by 15
- 2020Explaining Neural Networks by Decoding Layer Activationscited by 13
- 2020On the Information Plane of Autoencoderscited by 10
- 2020Are Deep Neural Architectures Losing Information? Invertibility Is Indispensablecited by 9
- 2022Explaining Classifiers by Constructing Familiar Conceptscited by 7
Built on by
Citing papers per year: 2025: 2.
2 papers.