# Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2402.14611 · HTML: https://mlsanity.com/paper/2402.14611/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [Iterative Normalization: Beyond Standardization towards Efficient Whitening](https://mlsanity.com/paper/1904.03441/) (2019)
- [DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation](https://mlsanity.com/paper/1604.02677/) (2016)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)

## Built on by (most cited first, 7 of 7)

- [Whitening Consistently Improves Self-Supervised Learning](https://mlsanity.com/paper/2408.07519/) (2024)
- [Saliency Assisted Quantization for Neural Networks](https://mlsanity.com/paper/2411.05858/) (2024)
- [Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)](https://mlsanity.com/paper/2403.18699/) (2024)
- [Mask & Match: Learning to Recognize Handwritten Math with Self-Supervised Attention](https://mlsanity.com/paper/2508.06107/) (2025)
- [Semi-Supervised Contrastive Learning with Orthonormal Prototypes](https://mlsanity.com/paper/2512.07880/) (2025)
- [SaliencyDecor: Enhancing Neural Network Interpretability through Feature Decorrelation](https://mlsanity.com/paper/2604.25315/) (2026)
- [Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks](https://mlsanity.com/paper/2605.18491/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) approaches have achieved great success when the amount of labeled data is limited. Within SSL, models learn robust feature representations by solving pretext tasks. One such pretext task is contrastive learning, which involves forming pairs of similar and dissimilar input samples, guiding the model to distinguish between them. In this work, we investigate the application of contrastive learning to the domain of medical image analysis. Our findings reveal that MoCo v2, a state-of-the-art contrastive learning method, encounters dimensional collapse when applied to medical images. This is attributed to the high degree of inter-image similarity shared between the medical images. To address this, we propose two key contributions: local feature learning and feature decorrelation. Local feature learning improves the ability of the model to focus on the local regions of the image, while feature decorrelation removes the linear dependence among the features. Our experimental findings demonstrate that our contributions significantly enhance the model's performance in the downstream task of medical segmentation, both in the linear evaluation and full fine-tuning settings. This work illustrates the importance of effectively adapting SSL techniques to the characteristics of medical imaging tasks. The source code will be made publicly available at: https://github.com/CAMMA-public/med-moco