mlsanityData to 7 Oct 2026

Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning

arXiv 2102.10680 ↗First posted 21 Feb 2021cs.CV eess.IVcited by 68

Not on the map. Why?

Abstract (arXiv)

This paper introduces a new concept called "transferable visual words" (TransVW), aiming to achieve annotation efficiency for deep learning in medical image analysis. Medical imaging--focusing on particular parts of the body for defined clinical purposes--generates images of great similarity in anatomy across patients and yields sophisticated anatomical patterns across images, which are associated with rich semantics about human anatomy and which are natural visual words. We show that these visual words can be automatically harvested according to anatomical consistency via self-discovery, and that the self-discovered visual words can serve as strong yet free supervision signals for deep models to learn semantics-enriched generic image representation via self-supervision (self-classification and self-restoration). Our extensive experiments demonstrate the annotation efficiency of TransVW by offering higher performance and faster convergence with reduced annotation cost in several applications. Our TransVW has several important advantages, including (1) TransVW is a fully autodidactic scheme, which exploits the semantics of visual words for self-supervised learning, requiring no expert annotation; (2) visual word learning is an add-on strategy, which complements existing self-supervised methods, boosting their performance; and (3) the learned image representation is semantics-enriched models, which have proven to be more robust and generalizable, saving annotation efforts for a variety of applications through transfer learning. Our code, pre-trained models, and curated visual words are available at https://github.com/JLiangLab/TransVW.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 21 Feb 2021.

Builds on

36 of 50 references.

References · 36

  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  2. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  3. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  5. 2014How transferable are features in deep neural networks?cited by 1,216
  6. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  7. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  8. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  9. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 935
  10. 2018Do Better ImageNet Models Transfer Better?cited by 560
  11. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  12. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
24 more
  1. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
  2. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  3. 2020What is being transferred in transfer learning?cited by 365
  4. 2020SCAN: Learning to Classify Images without Labelscited by 351
  5. 2020Contrastive learning of global and local features for medical image segmentation with limited annotationscited by 309
  6. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 282
  7. 2017Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?cited by 220
  8. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  9. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 198
  10. 2019Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysiscited by 159
  11. 2020Online Deep Clustering for Unsupervised Representation Learningcited by 99
  12. 2020How Useful is Self-Supervised Pretraining for Visual Tasks?cited by 80
  13. 2019AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Datacited by 77
  14. 2019Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cubecited by 76
  15. 2020Learning Representations by Predicting Bags of Visual Wordscited by 73
  16. 2019Self-Supervised Learning for Cardiac MR Image Segmentation by Anatomical Position Predictioncited by 70
  17. 2017NiftyNet: a deep-learning platform for medical imagingcited by 45
  18. 2019E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savingscited by 36
  19. 2020Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restorationcited by 35
  20. 2017Exploiting the potential of unlabeled endoscopic video data with self-supervised learningcited by 33
  21. 2017Improvements to context based self-supervised learningcited by 32
  22. 2020Steering Self-Supervised Feature Learning Beyond Local Pixel Statisticscited by 19
  23. 2018Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Predictioncited by 16
  24. 2015Exploiting Local Features from Deep Networks for Image Retrievalcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 13; 2023: 19; 2024: 14; 2025: 10; 2026: 5.

68 papers.

Sort:

By area:

  1. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  2. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  3. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  4. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  5. 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
  6. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  7. 2023LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matchingcited by 60
  8. 2022DiRA: Discriminative, Restorative, and Adversarial Learning for Self-supervised Medical Image Analysiscited by 48
  9. 2023PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysiscited by 40
  10. 2023Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-trainingcited by 36
  11. 2021SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detectioncited by 34
  12. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
38 more
  1. 2021A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysiscited by 31
  2. 2024Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomographycited by 31
  3. 2023Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysiscited by 25
  4. 2024Multi-modal Vision Pre-training for Medical Image Analysiscited by 22
  5. 2022Prompt Tuning for Parameter-efficient Medical Image Segmentationcited by 21
  6. 2024From Pixel to Cancer: Cellular Automata in Computed Tomographycited by 20
  7. 2023vox2vec: A Framework for Self-supervised Contrastive Learning of Voxel-level Representations in Medical Imagescited by 19
  8. 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
  9. 2022CAiD: Context-Aware Instance Discrimination for Self-supervised Learning in Medical Imagingcited by 18
  10. 2025CheXWorld: Exploring Image World Modeling for Radiograph Representation Learningcited by 18
  11. 2022UNetFormer: A Unified Vision Transformer Model and Pre-Training Framework for 3D Medical Image Segmentationcited by 16
  12. 2022Joint Self-Supervised Image-Volume Representation Learning with Intra-Inter Contrastive Clusteringcited by 16
  13. 2024Exploiting Structural Consistency of Chest Anatomy for Unsupervised Anomaly Detection in Radiography Imagescited by 13
  14. 2025Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Datacited by 13
  15. 2023HybridMIM: A Hybrid Masked Image Modeling Framework for 3D Medical Image Segmentationcited by 12
  16. 2024Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervisioncited by 12
  17. 2023Towards Foundation Models Learned from Anatomy in Medical Imaging via Self-Supervisioncited by 11
  18. 2026ReCo-KD: Region- and Context-Aware Knowledge Distillation for Efficient 3D Medical Image Segmentationcited by 10
  19. 2023AMAE: Adaptation of Pre-Trained Masked Autoencoder for Dual-Distribution Anomaly Detection in Chest X-Rayscited by 8
  20. 2025CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generationcited by 8
  21. 2024MambaMIM: Pre-training Mamba with State Space Token Interpolation and its Application to Medical Image Segmentationcited by 7
  22. 2021Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detectioncited by 6
  23. 2024Primitive Geometry Segment Pre-training for 3D Medical Image Segmentationcited by 6
  24. 2022Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directionscited by 5
  25. 2022Keep Your Friends Close & Enemies Farther: Debiasing Contrastive Learning with Spatial Priors in 3D Radiology Imagescited by 5
  26. 2024Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinementcited by 5
  27. 2021Self-supervised Tumor Segmentation through Layer Decompositioncited by 4
  28. 2023Pyramid Pixel Context Adaption Network for Medical Image Classification with Supervised Contrastive Learningcited by 4
  29. 2022A Point in the Right Direction: Vector Prediction for Spatially-aware Self-supervised Volumetric Representation Learningcited by 3
  30. 2023Localized Region Contrast for Enhancing Self-Supervised Learning in Medical Image Segmentationcited by 3
  31. 2024Symmetry Awareness Encoded Deep Learning Framework for Brain Imaging Analysiscited by 3
  32. 2025AFiRe: Anatomy-Driven Self-Supervised Learning for Fine-Grained Representation in Radiographic Imagescited by 3
  33. 2023Revisiting Fine-Tuning Strategies for Self-supervised Medical Imaging Analysiscited by 2
  34. 2023PET Synthesis via Self-supervised Adaptive Residual Estimation Generative Adversarial Networkcited by 2
  35. 2022Leveraging Global Binary Masks for Structure Segmentation in Medical Imagescited by 1
  36. 2023DrasCLR: A Self-supervised Framework of Learning Disease-related and Anatomy-specific Representation for 3D Medical Imagescited by 1
  37. 2024Cross-Dimensional Medical Self-Supervised Representation Learning Based on a Pseudo-3D Transformationcited by 1
  38. 2024Tissue-Contrastive Semi-Masked Autoencoders for Segmentation Pretraining on Chest CTcited by 1

50 most cited of 68 All 68 (JSON)