mlsanityData to 7 Oct 2026

A Global Geometric Analysis of Maximal Coding Rate Reduction

arXiv 2406.01909 ↗First posted 4 Jun 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

The maximal coding rate reduction (MCR2) objective for learning structured and compact deep representations is drawing increasing attention, especially after its recent usage in the derivation of fully explainable and highly effective deep network architectures. However, it lacks a complete theoretical justification: only the properties of its global optima are known, and its global landscape has not been studied. In this work, we give a complete characterization of the properties of all its local and global optima, as well as other types of critical points. Specifically, we show that each (local or global) maximizer of the MCR2 problem corresponds to a low-dimensional, discriminative, and diverse representation, and furthermore, each critical point of the objective is either a local maximizer or a strict saddle point. Such a favorable landscape makes MCR2 a natural choice of objective for learning diverse and discriminative representations via first-order optimization methods. To validate our theoretical findings, we conduct extensive experiments on both synthetic and real data sets.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

43 of 61 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2023Segment Anythingcited by 8,158
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  12. 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 411
31 more
  1. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  2. 2017How to Escape Saddle Points Efficientlycited by 284
  3. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  4. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  5. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  6. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 141
  7. 2020Neural collapse with unconstrained featurescited by 140
  8. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 127
  9. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  10. 2016A Geometric Analysis of Phase Retrievalcited by 92
  11. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  12. 2018Escaping Saddles with Stochastic Gradientscited by 66
  13. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  14. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  15. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  16. 2015Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picturecited by 49
  17. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  18. 2019Dimensionality compression and expansion in Deep Neural Networkscited by 38
  19. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  20. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 31
  21. 2017First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Timecited by 29
  22. 2020From Symmetry to Geometry: Tractable Nonconvex Problemscited by 28
  23. 2015When Are Nonconvex Problems Not Scary?cited by 27
  24. 2023Emergence of Segmentation with Minimalistic White-Box Transformerscited by 23
  25. 2023Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Modelscited by 22
  26. 2023White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?cited by 22
  27. 2021Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reductioncited by 17
  28. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  29. 2021Optimal Non-Convex Exact Recovery in Stochastic Block Model via Projected Power Methodcited by 10
  30. 2020Non-Convex Exact Community Recovery in Stochastic Block Modelcited by 5
  31. 2022Pursuit of a Discriminative Representation for Multiple Subspaces via Sequential Gamescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editingcited by 15
  2. 2025Exploring a Principled Framework for Deep Subspace Clusteringcited by 7
  3. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 5
  4. 2025Towards Interpretable and Efficient Attention: Compressing All by Contracting a Fewcited by 2
  5. 2025Linearly Separable Features in Shallow Nonlinear Networks: Width Scales Polynomially with Intrinsic Data Dimensioncited by 1
  6. 2025A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorizationcited by 1
  7. 2025Temporal Rate Reduction Clustering for Human Motion Segmentationcited by 1
  8. 2026Interpretable and Sparse Linear Attention with Decoupled Membership-Subspace Modeling via MCR2 Objectivecited by 0
  9. 2026Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discoverycited by 0
  10. 2026Operational Feature Fingerprints of Graph Datasets via a White-Box Signal-Subspace Probecited by 0
  11. 2026Jointly Learning Structured Representations and Stabilized Affinity for Human Motion Segmentationcited by 0
  12. 2026Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Informationcited by 0
1 more
  1. 2026On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisationcited by 0