mlsanityData to 14 Oct 2026

Learning to Compress: Local Rank and Information Compression in Deep Neural Networks

arXiv 2410.07687 ↗First posted 10 Oct 2024cs.LG cs.IT math.ITcited by 7

Not on the map. Why?

Abstract (arXiv)

Deep neural networks tend to exhibit a bias toward low-rank solutions during training, implicitly learning low-dimensional feature representations. This paper investigates how deep multilayer perceptrons (MLPs) encode these feature manifolds and connects this behavior to the Information Bottleneck (IB) theory. We introduce the concept of local rank as a measure of feature manifold dimensionality and demonstrate, both theoretically and empirically, that this rank decreases during the final phase of training. We argue that networks that reduce the rank of their learned representations also compress mutual information between inputs and intermediate layers. This work bridges the gap between feature manifold rank and information compression, offering new insights into the interplay between information bottlenecks and representation learning.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

19 of 26 references.

References · 19

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  2. 2016Deep Variational Information Bottleneckcited by 930
  3. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  4. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  6. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  7. 2017Implicit Regularization in Matrix Factorizationcited by 322
  8. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  9. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  10. 2020Directional convergence and alignment in deep learningcited by 178
  11. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  12. 2022Rank Diminishing in Deep Neural Networkscited by 47
7 more
  1. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  2. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  3. 2023Reverse Engineering Self-Supervised Learningcited by 33
  4. 2022Information Flow in Deep Neural Networkscited by 22
  5. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  6. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  7. 2024What Secrets Do Your Manifolds Hold? Understanding the Local Geometry of Generative Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  2. 2026Pointwise Generalization in Deep Neural Networkscited by 4
  3. 2026JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physicscited by 3
  4. 2025Training Large Neural Networks With Low-Dimensional Error Feedbackcited by 2
  5. 2026Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Interventioncited by 2
  6. 2025Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognitioncited by 0
  7. 2026Geometric and Information Compression of Representations in Deep Learningcited by 0