# vox2vec: A Framework for Self-supervised Contrastive Learning of Voxel-level Representations in Medical Images

arXiv: https://arxiv.org/abs/2307.14725 · HTML: https://mlsanity.com/paper/2307.14725/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2011.10043/) (2020)
- [Unsupervised Learning of Dense Visual Representations](https://mlsanity.com/paper/2011.05499/) (2020)
- [3D Self-Supervised Methods for Medical Imaging](https://mlsanity.com/paper/2006.03829/) (2020)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning](https://mlsanity.com/paper/2102.10680/) (2021)
- [SAM: Self-supervised Learning of Pixel-wise Anatomical Embeddings in Radiological Images](https://mlsanity.com/paper/2012.02383/) (2020)
- [Voxel-level Siamese Representation Learning for Abdominal Multi-Organ Segmentation](https://mlsanity.com/paper/2105.07672/) (2021)

## Built on by (most cited first, 19 of 19)

- [Revisiting MAE pre-training for 3D medical image segmentation](https://mlsanity.com/paper/2410.23132/) (2024)
- [T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency](https://mlsanity.com/paper/2312.01529/) (2023)
- [HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Training](https://mlsanity.com/paper/2408.05815/) (2024)
- [Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation](https://mlsanity.com/paper/2406.07146/) (2024)
- [CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generation](https://mlsanity.com/paper/2508.05375/) (2025)
- [MambaMIM: Pre-training Mamba with State Space Token Interpolation and its Application to Medical Image Segmentation](https://mlsanity.com/paper/2408.08070/) (2024)
- [Real Time Multi Organ Classification on Computed Tomography Images](https://mlsanity.com/paper/2404.18731/) (2024)
- [Dense Self-Supervised Learning for Medical Image Segmentation](https://mlsanity.com/paper/2407.20395/) (2024)
- [Anatomical Positional Embeddings](https://mlsanity.com/paper/2409.10291/) (2024)
- [Self-supervised Learning of Dense Hierarchical Representations for Medical Image Segmentation](https://mlsanity.com/paper/2401.06473/) (2024)
- [Cross-Dimensional Medical Self-Supervised Representation Learning Based on a Pseudo-3D Transformation](https://mlsanity.com/paper/2406.00947/) (2024)
- [Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models](https://mlsanity.com/paper/2509.09064/) (2025)
- [AMLP: Adjustable Masking Lesion Patches for Self-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2309.04312/) (2023)
- [Screener: Self-supervised Pathology Segmentation in Medical CT Images](https://mlsanity.com/paper/2502.08321/) (2025)
- [BodyGPS: Anatomical Positioning System](https://mlsanity.com/paper/2505.07744/) (2025)
- [CoRe: Joint Optimization with Contrastive Learning for Medical Image Registration](https://mlsanity.com/paper/2603.23694/) (2026)
- [Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines](https://mlsanity.com/paper/2605.23995/) (2026)
- [Contrastive Learning for Seismic Horizon Tracking with Domain-Specific Priors](https://mlsanity.com/paper/2606.16271/) (2026)
- [Pix2Rep-v2: Data-Efficient Representation Learning for Dense Medical Imaging Applications](https://mlsanity.com/paper/2609.01427/) (2026)

## Abstract (arXiv)

This paper introduces vox2vec - a contrastive method for self-supervised learning (SSL) of voxel-level representations. vox2vec representations are modeled by a Feature Pyramid Network (FPN): a voxel representation is a concatenation of the corresponding feature vectors from different pyramid levels. The FPN is pre-trained to produce similar representations for the same voxel in different augmented contexts and distinctive representations for different voxels. This results in unified multi-scale representations that capture both global semantics (e.g., body part) and local semantics (e.g., different small organs or healthy versus tumor tissue). We use vox2vec to pre-train a FPN on more than 6500 publicly available computed tomography images. We evaluate the pre-trained representations by attaching simple heads on top of them and training the resulting models for 22 segmentation tasks. We show that vox2vec outperforms existing medical imaging SSL techniques in three evaluation setups: linear and non-linear probing and end-to-end fine-tuning. Moreover, a non-linear head trained on top of the frozen vox2vec representations achieves competitive performance with the FPN trained from scratch while having 50 times fewer trainable parameters. The code is available at https://github.com/mishgon/vox2vec .