mlsanityData to 7 Oct 2026

LDReg: Local Dimensionality Regularized Self-Supervised Learning

arXiv 2401.10474 ↗First posted 19 Jan 2024cs.LG cs.AI cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Representations learned via self-supervised learning (SSL) can be susceptible to dimensional collapse, where the learned representation subspace is of extremely low dimensionality and thus fails to represent the full data distribution and modalities. Dimensional collapse also known as the "underfilling" phenomenon is one of the major causes of degraded performance on downstream tasks. Previous work has investigated the dimensional collapse problem of SSL at a global level. In this paper, we demonstrate that representations can span over high dimensional space globally, but collapse locally. To address this, we propose a method called textitlocal dimensionality regularization (LDReg). Our formulation is based on the derivation of the Fisher-Rao metric to compare and optimize local distance distributions at an asymptotically small radius for each data point. By increasing the local intrinsic dimensionality, we demonstrate through a range of experiments that LDReg improves the representation quality of SSL. The results also show that LDReg can regularize dimensionality at both local and global levels.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 46 of its 73 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

46 of 73 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,853
34 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  8. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  12. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  13. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  14. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  15. 2018Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionalitycited by 280
  16. 2020Whitening for Self-Supervised Representation Learningcited by 278
  17. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  18. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  19. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  20. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  21. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  22. 2021Decoupled Contrastive Learningcited by 141
  23. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  24. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  25. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  26. 2018On the Intrinsic Dimensionality of Image Representationscited by 57
  27. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  28. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  29. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  30. 2022Unsupervised Visual Representation Learning by Synchronous Momentum Groupingcited by 22
  31. 2023Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learningcited by 18
  32. 2022Toward a Geometrical Understanding of Self-supervised Contrastive Learningcited by 17
  33. 2022On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learningcited by 16
  34. 2022COLLIDER: A Robust Training Framework for Backdoor Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 4.

11 papers.

  1. 2025Measuring the Intrinsic Dimension of Earth Representationscited by 10
  2. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  3. 2024DDA: Dimensionality Driven Augmentation Search for Contrastive Learning in Laparoscopic Surgerycited by 3
  4. 2024Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attackscited by 3
  5. 2026Global Geometry Is Not Enough for Vision Representationscited by 2
  6. 2024POA: Pre-training Once for Models of All Sizescited by 1
  7. 2025Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimensioncited by 1
  8. 2025CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignmentcited by 0
  9. 2026On the Use of Bagging for Local Intrinsic Dimensionality Estimationcited by 0
  10. 2026When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compressioncited by 0
  11. 2026Learning a Flow to Self-Supervised Representationscited by 0