mlsanityData to 7 Oct 2026

ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction

arXiv 2105.10446 ↗First posted 21 May 2021cs.LG cs.CV cs.ITcited by 84

Not on the map. Why?

Abstract (arXiv)

This work attempts to provide a plausible theoretical framework that aims to interpret modern deep (convolutional) networks from the principles of data compression and discriminative representation. We argue that for high-dimensional multi-class data, the optimal linear discriminative representation maximizes the coding rate difference between the whole dataset and the average of all the subsets. We show that the basic iterative gradient ascent scheme for optimizing the rate reduction objective naturally leads to a multi-layer deep network, named ReduNet, which shares common characteristics of modern deep networks. The deep layered architectures, linear and nonlinear operators, and even parameters of the network are all explicitly constructed layer-by-layer via forward propagation, although they are amenable to fine-tuning via back propagation. All components of so-obtained "white-box" network have precise optimization, statistical, and geometric interpretation. Moreover, all linear operators of the so-derived network naturally become multi-channel convolutions when we enforce classification to be rigorously shift-invariant. The derivation in the invariant setting suggests a trade-off between sparsity and invariance, and also indicates that such a deep convolution network is significantly more efficient to construct and learn in the spectral domain. Our preliminary simulations and experiments clearly verify the effectiveness of both the rate reduction objective and the associated ReduNet. All code and data are available at urlhttps://github.com/Ma-Lab-Berkeley.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 37 of its 175 references that are in the map. First posted on arXiv 21 May 2021.

Builds on

37 of 175 references.

References · 37

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2016Layer Normalizationcited by 5,579
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  11. 2016Group Equivariant Convolutional Networkscited by 1,154
  12. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
25 more
  1. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  2. 2017Dynamic Routing Between Capsulescited by 689
  3. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  4. 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 411
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  6. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  7. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  8. 2015Highway Networkscited by 257
  9. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  10. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  11. 2018Activation Functions: Comparison of trends in Practice and Research for Deep Learningcited by 130
  12. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  13. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  14. 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
  15. 2017The power of deeper networks for expressing natural functionscited by 35
  16. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  17. 2018Caveats for information bottleneck in deterministic scenarioscited by 34
  18. 2014Fast Convolutional Nets With fbfft: A GPU Performance Evaluationcited by 27
  19. 2019A Rate-Distortion Framework for Explaining Neural Network Decisionscited by 24
  20. 2019Neural Collaborative Subspace Clusteringcited by 21
  21. 2019Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigmcited by 20
  22. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  23. 2016Interpretable Recurrent Neural Networks Using Sequential Sparse Recoverycited by 8
  24. 2017Deep Sparse Subspace Clusteringcited by 7
  25. 2015LogDet Rank Minimization with Application to Subspace Clusteringcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 18; 2023: 18; 2024: 20; 2025: 12; 2026: 9.

84 papers.

Sort:

By area:

  1. 2022Discovering Invariant Rationales for Graph Neural Networkscited by 186
  2. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  3. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  4. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  5. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  6. 2021How Powerful is Graph Convolution for Recommendation?cited by 50
  7. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  8. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  9. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  10. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  11. 2022Green Learning: Introduction, Examples and Outlookcited by 30
  12. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
38 more
  1. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  2. 2023Emergence of Segmentation with Minimalistic White-Box Transformerscited by 23
  3. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  4. 2021Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reductioncited by 17
  5. 2022Incremental Learning of Structured Memory via Closed-Loop Transcriptioncited by 17
  6. 2021MAE-DET: Revisiting Maximum Entropy Principle in Zero-Shot NAS for Efficient Object Detectioncited by 16
  7. 2021Learning High-Dimensional Parametric Maps via Reduced Basis Adaptive Residual Networkscited by 16
  8. 2022Learning Fair Representations via Rate-Distortion Maximizationcited by 15
  9. 2024A Law of Next-Token Prediction in Large Language Modelscited by 15
  10. 2023DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Networkcited by 14
  11. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  12. 2025Data Pruning by Information Maximizationcited by 14
  13. 2023Representation Learning via Manifold Flattening and Reconstructioncited by 11
  14. 2023Data Representations' Study of Latent Image Manifoldscited by 11
  15. 2023Towards Task Sampler Learning for Meta-Learningcited by 11
  16. 2024Speculative Coreset Selection for Task-Specific Fine-tuningcited by 10
  17. 2024Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and Levels-of-Expertscited by 9
  18. 2024Masked Completion via Structured Diffusion with White-Box Transformerscited by 9
  19. 2022Unsupervised Learning of Structured Representations via Closed-Loop Transcriptioncited by 7
  20. 2023Threshold-Consistent Margin Loss for Open-World Deep Metric Learningcited by 7
  21. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6
  22. 2023Data Optimization in Deep Learning: A Surveycited by 6
  23. 2022Sustaining Fairness via Incremental Learningcited by 5
  24. 2022Unsupervised Opinion Summarization Using Approximate Geodesicscited by 5
  25. 2022Extracting Semantic Knowledge from GANs with Unsupervised Learningcited by 5
  26. 2023Maximizing Spatio-Temporal Entropy of Deep 3D CNNs for Efficient Video Recognitioncited by 5
  27. 2023Achieving Robust Generalization for Wireless Channel Estimation Neural Networks by Designed Training Datacited by 4
  28. 2023Closed-Loop Transcription via Convolutional Sparse Codingcited by 4
  29. 2021Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristicscited by 3
  30. 2022Bridging Model-based Safety and Model-free Reinforcement Learning through System Identification of Low Dimensional Linear Modelscited by 3
  31. 2022UGformer for Robust Left Atrium and Scar Segmentation Across Scannerscited by 3
  32. 2024Merino: Entropy-driven Design for Generative Language Models on IoT Devicescited by 3
  33. 2024LightFF: Lightweight Inference for Forward-Forward Algorithmcited by 3
  34. 2024Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendationcited by 3
  35. 2024Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspectivecited by 3
  36. 2025Geometry of Neural Reinforcement Learning in Continuous State and Action Spacescited by 3
  37. 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
  38. 2022Augmentation Invariant Manifold Learningcited by 2

50 most cited of 84 All 84 (JSON)