# Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restoration

arXiv: https://arxiv.org/abs/2007.06959 · HTML: https://mlsanity.com/paper/2007.06959/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 30 of 35)

- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning](https://mlsanity.com/paper/2102.10680/) (2021)
- [Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification](https://mlsanity.com/paper/2210.12843/) (2022)
- [UACANet: Uncertainty Augmented Context Attention for Polyp Segmentation](https://mlsanity.com/paper/2107.02368/) (2021)
- [Preservational Learning Improves Self-supervised Medical Image Models by Reconstructing Diverse Contexts](https://mlsanity.com/paper/2109.04379/) (2021)
- [OpenForensics: Large-Scale Challenging Dataset For Multi-Face Forgery Detection And Segmentation In-The-Wild](https://mlsanity.com/paper/2107.14480/) (2021)
- [DiRA: Discriminative, Restorative, and Adversarial Learning for Self-supervised Medical Image Analysis](https://mlsanity.com/paper/2204.10437/) (2022)
- [Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-training](https://mlsanity.com/paper/2303.00874/) (2023)
- [Self-supervised 3D anatomy segmentation using self-distilled masked image transformer (SMIT)](https://mlsanity.com/paper/2205.10342/) (2022)
- [A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis](https://mlsanity.com/paper/2108.05930/) (2021)
- [Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works](https://mlsanity.com/paper/2211.08129/) (2022)
- [How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?](https://mlsanity.com/paper/2501.11253/) (2025)
- [CAiD: Context-Aware Instance Discrimination for Self-supervised Learning in Medical Imaging](https://mlsanity.com/paper/2204.07344/) (2022)
- [A Location-Sensitive Local Prototype Network for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2103.10178/) (2021)
- [Label-Efficient Deep Learning in Medical Image Analysis: Challenges and Future Directions](https://mlsanity.com/paper/2303.12484/) (2023)
- [Exploiting Structural Consistency of Chest Anatomy for Unsupervised Anomaly Detection in Radiography Images](https://mlsanity.com/paper/2403.08689/) (2024)
- [Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision](https://mlsanity.com/paper/2404.15672/) (2024)
- [Analyzing Tumors by Synthesis](https://mlsanity.com/paper/2409.06035/) (2024)
- [Anatomy-Aware Contrastive Representation Learning for Fetal Ultrasound](https://mlsanity.com/paper/2208.10642/) (2022)
- [Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection](https://mlsanity.com/paper/2109.07029/) (2021)
- [Data efficient deep learning for medical image analysis: A survey](https://mlsanity.com/paper/2310.06557/) (2023)
- [Unsupervised Local Discrimination for Medical Images](https://mlsanity.com/paper/2108.09440/) (2021)
- [Advancing 3D Medical Image Analysis with Variable Dimension Transform based Supervised 3D Pre-training](https://mlsanity.com/paper/2201.01426/) (2022)
- [A Survey of the Impact of Self-Supervised Pretraining for Diagnostic Tasks with Radiological Images](https://mlsanity.com/paper/2309.02555/) (2023)
- [PET Synthesis via Self-supervised Adaptive Residual Estimation Generative Adversarial Network](https://mlsanity.com/paper/2310.15550/) (2023)
- [Transfer Learning for Segmentation Problems: Choose the Right Encoder and Skip the Decoder](https://mlsanity.com/paper/2207.14508/) (2022)
- [Consecutive Knowledge Meta-Adaptation Learning for Unsupervised Medical Diagnosis](https://mlsanity.com/paper/2209.10425/) (2022)
- [Unsupervised Feature Clustering Improves Contrastive Representation Learning for Medical Image Segmentation](https://mlsanity.com/paper/2211.08557/) (2022)
- [ACE: Anatomically Consistent Embeddings in Composition and Decomposition](https://mlsanity.com/paper/2501.10131/) (2025)

## Abstract (arXiv)

Medical images are naturally associated with rich semantics about the human anatomy, reflected in an abundance of recurring anatomical patterns, offering unique potential to foster deep semantic representation learning and yield semantically more powerful models for different medical applications. But how exactly such strong yet free semantics embedded in medical images can be harnessed for self-supervised learning remains largely unexplored. To this end, we train deep models to learn semantically enriched visual representation by self-discovery, self-classification, and self-restoration of the anatomy underneath medical images, resulting in a semantics-enriched, general-purpose, pre-trained 3D model, named Semantic Genesis. We examine our Semantic Genesis with all the publicly-available pre-trained models, by either self-supervision or fully supervision, on the six distinct target tasks, covering both classification and segmentation in various medical modalities (i.e.,CT, MRI, and X-ray). Our extensive experiments demonstrate that Semantic Genesis significantly exceeds all of its 3D counterparts as well as the de facto ImageNet-based transfer learning in 2D. This performance is attributed to our novel self-supervised learning framework, encouraging deep models to learn compelling semantic representation from abundant anatomical patterns resulting from consistent anatomies embedded in medical images. Code and pre-trained Semantic Genesis are available at https://github.com/JLiangLab/SemanticGenesis .