mlsanityData to 14 Oct 2026

Complexity of Linear Regions in Self-supervised Deep ReLU Networks

arXiv 2604.24393 ↗First posted 27 Apr 2026cs.LG cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

There has been growing interest in studying the complexity of Rectified Linear Unit (ReLU) based activation networks. Recent work investigates the evolution of the number of piecewise-linear partitions (linear regions) that are formed during training. However, current research is limited to examining the complexity of models trained in a supervised way. Self-Supervised Learning (SSL) differs in that it directly optimises the representation space using a loss function to enhance the model's performance across multiple downstream tasks. This study investigates the local distribution of linear regions produced by SSL models. We demonstrate that the evolution of linear regions correlates with the representation quality by utilising SplineCam to extract two-dimensional polytopes near the data distribution. We track the number, area, eccentricity, and boundaries of regions throughout training. The study compares supervised, contrastive, and self-distillation methods over two standard benchmark datasets, MNIST and FashionMNIST. The analysis of the experimental results shows that self-supervised methods create substantially fewer regions to achieve comparable accuracy to supervised models. Contrastive methods rapidly expand regions over time, whereas self-distillation methods tend to consolidate by merging neighbouring regions. Lastly, we can detect representation collapse early within the geometric space of linear regions. Our analysis suggests that polytopal metrics can serve as reliable indicators of representation quality and model performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 34 of its 41 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

34 of 41 references.

References · 34

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2020Supervised Contrastive Learningcited by 2,829
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  11. 2025DINOv3cited by 1,414
  12. 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 786
22 more
  1. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  2. 2016On the Expressive Power of Deep Neural Networkscited by 306
  3. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  4. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  5. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  6. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
  7. 2019Complexity of Linear Regions in Deep Networkscited by 167
  8. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  9. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  10. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  11. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  12. 2018Provable Robustness of ReLU networks via Maximization of Linear Regionscited by 51
  13. 2024Deep Networks Always Grok and Here is Whycited by 51
  14. 2017The power of deeper networks for expressing natural functionscited by 35
  15. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  16. 2022Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Valuescited by 21
  17. 2022Are All Linear Regions Created Equal?cited by 13
  18. 2021Test Sample Accuracy Scales with Training Sample Density in Neural Networkscited by 9
  19. 2021Self-Supervised Features Improve Open-World Learningcited by 8
  20. 2024The Common Stability Mechanism behind most Self-Supervised Learning Approachescited by 8
  21. 2024What Secrets Do Your Manifolds Hold? Understanding the Local Geometry of Generative Modelscited by 7
  22. 2025Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Timecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Self-Distilled Self-Supervised Representation Learning
  2. 2024Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?
  3. 2022Effective Self-supervised Pre-training on Low-compute Networks without Distillation
  4. 2024A Review on Discriminative Self-supervised Learning Methods in Computer Vision
  5. 2023Unifying Synergies between Self-supervised Learning and Dynamic Computation
  6. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Models
  7. 2024On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning
  8. 2026REALM: Regime-Switching, Explainable, and Activation-Induced Linear Models