# Medical Slice Transformer: Improved Diagnosis and Explainability on 3D Medical Images with DINOv2

arXiv: https://arxiv.org/abs/2411.15802 · HTML: https://mlsanity.com/paper/2411.15802/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 24 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning](https://mlsanity.com/paper/2003.04696/) (2020)
- [An Overview of Deep Semi-Supervised Learning](https://mlsanity.com/paper/2006.05278/) (2020)
- [Explainable deep learning models in medical image analysis](https://mlsanity.com/paper/2005.13799/) (2020)
- [Are Natural Domain Foundation Models Useful for Medical Image Classification?](https://mlsanity.com/paper/2310.19522/) (2023)
- [How Transferable Are Self-supervised Features in Medical Image Classification Tasks?](https://mlsanity.com/paper/2108.10048/) (2021)
- [Going Deep in Medical Image Analysis: Concepts, Methods, Challenges and Future Directions](https://mlsanity.com/paper/1902.05655/) (2019)
- [Towards Evaluating Explanations of Vision Transformers for Medical Imaging](https://mlsanity.com/paper/2304.06133/) (2023)
- [AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation](https://mlsanity.com/paper/2202.06431/) (2022)
- [Comparative Analysis of ImageNet Pre-Trained Deep Learning Models and DINOv2 in Medical Imaging Classification](https://mlsanity.com/paper/2402.07595/) (2024)
- [Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification](https://mlsanity.com/paper/2303.01871/) (2023)
- [Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images](https://mlsanity.com/paper/2308.07688/) (2023)
- [Self-supervision for medical image classification: state-of-the-art performance with ~100 labeled training samples per class](https://mlsanity.com/paper/2304.05163/) (2023)
- [DINO-CXR: A self supervised method based on vision transformer for chest X-ray classification](https://mlsanity.com/paper/2308.00475/) (2023)
- [SE3D: A Framework For Saliency Method Evaluation In 3D Imaging](https://mlsanity.com/paper/2405.14584/) (2024)

## Built on by (most cited first, 13 of 13)

- [Revisiting 2D Foundation Models for Scalable 3D Medical Image Classification](https://mlsanity.com/paper/2512.12887/) (2025)
- [A European Multi-Center Breast Cancer MRI Dataset](https://mlsanity.com/paper/2506.00474/) (2025)
- [Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registration](https://mlsanity.com/paper/2509.06467/) (2025)
- [MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis](https://mlsanity.com/paper/2509.06617/) (2025)
- [Structured Spectral Graph Representation Learning for Multi-label Abnormality Analysis from 3D CT Scans](https://mlsanity.com/paper/2510.10779/) (2025)
- [Anatomy-Aware Low-Dose CT Denoising via Pretrained Vision Models and Semantic-Guided Contrastive Learning](https://mlsanity.com/paper/2508.07788/) (2025)
- [Toward a robust lesion detection model in breast DCE-MRI: adapting foundation models to high-risk women](https://mlsanity.com/paper/2509.02710/) (2025)
- [Multi Anatomy X-Ray Foundation Model](https://mlsanity.com/paper/2509.12146/) (2025)
- [Masked and Predictive Self-Supervised Foundation Models for 3D Brain MRI](https://mlsanity.com/paper/2606.13315/) (2026)
- [Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography](https://mlsanity.com/paper/2607.26196/) (2026)
- [Adapted Foundation Models for Breast MRI Triaging in Contrast-Enhanced and Non-Contrast Enhanced Protocols](https://mlsanity.com/paper/2511.05967/) (2025)
- [DINO-3DRA: Leveraging 2D Foundation Model Semantics for 3D Cerebral Aneurysm Segmentation](https://mlsanity.com/paper/2608.07767/) (2026)
- [Faithful Faithfulness Evaluations: Challenges & Pitfalls Learned from a Breast MRI Case Study](https://mlsanity.com/paper/2609.25978/) (2026)

## Abstract (arXiv)

MRI and CT are essential clinical cross-sectional imaging techniques for diagnosing complex conditions. However, large 3D datasets with annotations for deep learning are scarce. While methods like DINOv2 are encouraging for 2D image analysis, these methods have not been applied to 3D medical images. Furthermore, deep learning models often lack explainability due to their "black-box" nature. This study aims to extend 2D self-supervised models, specifically DINOv2, to 3D medical imaging while evaluating their potential for explainable outcomes. We introduce the Medical Slice Transformer (MST) framework to adapt 2D self-supervised models for 3D medical image analysis. MST combines a Transformer architecture with a 2D feature extractor, i.e., DINOv2. We evaluate its diagnostic performance against a 3D convolutional neural network (3D ResNet) across three clinical datasets: breast MRI (651 patients), chest CT (722 patients), and knee MRI (1199 patients). Both methods were tested for diagnosing breast cancer, predicting lung nodule dignity, and detecting meniscus tears. Diagnostic performance was assessed by calculating the Area Under the Receiver Operating Characteristic Curve (AUC). Explainability was evaluated through a radiologist's qualitative comparison of saliency maps based on slice and lesion correctness. P-values were calculated using Delong's test. MST achieved higher AUC values compared to ResNet across all three datasets: breast (0.94±0.01 vs. 0.91±0.02, P=0.02), chest (0.95±0.01 vs. 0.92±0.02, P=0.13), and knee (0.85±0.04 vs. 0.69±0.05, P=0.001). Saliency maps were consistently more precise and anatomically correct for MST than for ResNet. Self-supervised 2D models like DINOv2 can be effectively adapted for 3D medical imaging using MST, offering enhanced diagnostic accuracy and explainability compared to convolutional neural networks.