# Structure-aware Semantic Discrepancy and Consistency for 3D Medical Image Self-supervised Learning

arXiv: https://arxiv.org/abs/2507.02581 · HTML: https://mlsanity.com/paper/2507.02581/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 30 of 34)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [Generative Adversarial Network in Medical Imaging: A Review](https://mlsanity.com/paper/1809.07294/) (2018)
- [Neighbourhood Consensus Networks](https://mlsanity.com/paper/1810.10510/) (2018)
- [3D Self-Supervised Methods for Medical Imaging](https://mlsanity.com/paper/2006.03829/) (2020)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysis](https://mlsanity.com/paper/2402.17300/) (2024)
- [Masked Image Modeling Advances 3D Medical Image Analysis](https://mlsanity.com/paper/2204.11716/) (2022)
- [Preservational Learning Improves Self-supervised Medical Image Models by Reconstructing Diverse Contexts](https://mlsanity.com/paper/2109.04379/) (2021)
- [Truly shift-invariant convolutional neural networks](https://mlsanity.com/paper/2011.14214/) (2020)
- [Revisiting Rubik's Cube: Self-supervised Learning with Volume-wise Transformation for 3D Medical Image Segmentation](https://mlsanity.com/paper/2007.08826/) (2020)
- [Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-training](https://mlsanity.com/paper/2303.00874/) (2023)
- [SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2307.12591/) (2023)
- [Jigsaw Clustering for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2104.00323/) (2021)
- [Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysis](https://mlsanity.com/paper/2302.05615/) (2023)
- [GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding](https://mlsanity.com/paper/2403.09639/) (2024)
- [Advancing Volumetric Medical Image Segmentation via Global-Local Masked Autoencoder](https://mlsanity.com/paper/2306.08913/) (2023)
- [MiM: Mask in Mask Self-Supervised Pre-Training for 3D Medical Image Analysis](https://mlsanity.com/paper/2404.15580/) (2024)
- [Robust Perception through Equivariance](https://mlsanity.com/paper/2212.06079/) (2022)
- [Attentive Symmetric Autoencoder for Brain MRI Segmentation](https://mlsanity.com/paper/2209.08887/) (2022)

## Built on by (most cited first, 5 of 5)

- [Minimal Semantic Sufficiency Meets Unsupervised Domain Generalization](https://mlsanity.com/paper/2509.15791/) (2025)
- [Exploiting Completeness Perception with Diffusion Transformer for Unified 3D MRI Synthesis](https://mlsanity.com/paper/2602.18400/) (2026)
- [Adaptive Texture-aware Masking for Self-Supervised Learning in 3D Dental CBCT Analysis](https://mlsanity.com/paper/2605.01741/) (2026)
- [Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging](https://mlsanity.com/paper/2605.14654/) (2026)
- [Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning](https://mlsanity.com/paper/2610.13915/) (2026)

## Abstract (arXiv)

3D medical image self-supervised learning (mSSL) holds great promise for medical analysis. Effectively supporting broader applications requires considering anatomical structure variations in location, scale, and morphology, which are crucial for capturing meaningful distinctions. However, previous mSSL methods partition images with fixed-size patches, often ignoring the structure variations. In this work, we introduce a novel perspective on 3D medical images with the goal of learning structure-aware representations. We assume that patches within the same structure share the same semantics (semantic consistency) while those from different structures exhibit distinct semantics (semantic discrepancy). Based on this assumption, we propose an mSSL framework named S2DC, achieving Structure-aware Semantic Discrepancy and Consistency in two steps. First, S2DC enforces distinct representations for different patches to increase semantic discrepancy by leveraging an optimal transport strategy. Second, S2DC advances semantic consistency at the structural level based on neighborhood similarity distribution. By bridging patch-level and structure-level representations, S2DC achieves structure-aware representations. Thoroughly evaluated across 10 datasets, 4 tasks, and 3 modalities, our proposed method consistently outperforms the state-of-the-art methods in mSSL.