mlsanityData to 7 Oct 2026

End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training

arXiv 2402.09050 ↗First posted 14 Feb 2024cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

End-to-end (E2E) training, optimizing the entire model through error backpropagation, fundamentally supports the advancements of deep learning. Despite its high performance, E2E training faces the problems of memory consumption, parallel computing, and discrepancy with the functionalities of the actual brain. Various alternative methods have been proposed to overcome these difficulties; however, no one can yet match the performance of E2E training, thereby falling short in practicality. Furthermore, there is no deep understanding regarding differences in the trained model properties beyond the performance gap. In this paper, we reconsider why E2E training demonstrates a superior performance through a comparison with layer-wise training, a non-E2E method that locally sets errors. On the basis of the observation that E2E training has an advantage in propagating input information, we analyze the information plane dynamics of intermediate representations based on the Hilbert-Schmidt independence criterion (HSIC). The results of our normalized HSIC value analysis reveal the E2E training ability to exhibit different information dynamics across layers, in addition to efficient information propagation. Furthermore, we show that this layer-role differentiation leads to the final representation following the information bottleneck principle. It suggests the need to consider the cooperative interactions between layers, not just the final layer when analyzing the information bottleneck of deep learning.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 66 of its 97 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

66 of 97 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2020Supervised Contrastive Learningcited by 2,803
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  11. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 825
38 more
  1. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  2. 2016Universal adversarial perturbationscited by 774
  3. 2020What is the State of Neural Network Pruning?cited by 578
  4. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  5. 2019On Mutual Information Maximization for Representation Learningcited by 315
  6. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  7. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  8. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  9. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  10. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  11. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  12. 2014Difference Target Propagationcited by 144
  13. 2019Training Neural Networks with Local Error Signalscited by 135
  14. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  15. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
  16. 2019Analyzing and Improving Representations with the Soft Nearest Neighbor Losscited by 95
  17. 2019Are All Layers Created Equal?cited by 94
  18. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  19. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  20. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 77
  21. 2019The HSIC Bottleneck: Deep Learning without Back-Propagationcited by 77
  22. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  23. 2019Decoupled Greedy Learning of CNNscited by 76
  24. 2022Gradients without Backpropagationcited by 73
  25. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  26. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  27. 2022Hebbian Deep Learning Without Feedbackcited by 61
  28. 2020LoCo: Local Contrastive Representation Learningcited by 55
  29. 2014Random feedback weights support learning in deep neural networkscited by 47
  30. 2017Deep supervised learning using local errorscited by 41
  31. 2020On Information Plane Analyses of Neural Network Classifiers -- A Reviewcited by 35
  32. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  33. 2017Learning Deep ResNet Blocks Sequentially using Boosting Theorycited by 32
  34. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 31
  35. 2021SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networkscited by 29
  36. 2020Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networkscited by 25
  37. 2022Signal Propagation: A Framework for Learning and Inference In a Forward Passcited by 24
  38. 2023Blockwise Self-Supervised Learning at Scalecited by 22

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 1.

8 papers.

  1. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  2. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  3. 2025Enhanced Structured Lasso Pruning with Class-wise Informationcited by 1
  4. 2025AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networkscited by 1
  5. 2024Real Time American Sign Language Detection Using Yolo-v9cited by 0
  6. 2025Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognitioncited by 0
  7. 2025LightHCG: a Lightweight yet powerful HSIC Disentanglement based Causal Glaucoma Detection Model frameworkcited by 0
  8. 2026Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformerscited by 0