# DALE-CT: Depth-Aware 2D Slice Encoders Learn an Anatomical World Model of Chest CT

arXiv: https://arxiv.org/abs/2606.07775 · HTML: https://mlsanity.com/paper/2606.07775/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [Exploring scalable medical image encoders beyond text supervision](https://mlsanity.com/paper/2401.10815/) (2024)
- [Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset](https://mlsanity.com/paper/2406.06512/) (2024)
- [Vision Foundation Models for Computed Tomography](https://mlsanity.com/paper/2501.09001/) (2025)
- [Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks](https://mlsanity.com/paper/2312.02366/) (2023)
- [When Does LeJEPA Learn a World Model?](https://mlsanity.com/paper/2605.26379/) (2026)
- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [ReXGroundingCT: A 3D Chest CT Dataset for Segmentation of Findings from Free-Text Reports](https://mlsanity.com/paper/2507.22030/) (2025)
- [U-VLM: Hierarchical Vision Language Modeling for Report Generation](https://mlsanity.com/paper/2603.00479/) (2026)
- [Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression](https://mlsanity.com/paper/2603.23308/) (2026)
- [DINO-MX: A Modular & Flexible Framework for Self-Supervised Learning](https://mlsanity.com/paper/2511.01610/) (2025)
- [TAP-CT: 3D Task-Agnostic Pretraining of Computed Tomography Foundation Models](https://mlsanity.com/paper/2512.00872/) (2025)
- [Organ-Aware Attention Improves CT Triage and Classification](https://mlsanity.com/paper/2601.13385/) (2026)
- [DINO-LG: Enhancing Vision Transformers with Label Guidance for Coronary Artery Calcium Detection](https://mlsanity.com/paper/2411.07976/) (2024)
- [A Framework for Cross-Domain Generalization in Coronary Artery Calcium Scoring Across Gated and Non-Gated Computed Tomography](https://mlsanity.com/paper/2602.21935/) (2026)
- [JANUS: Anatomy-Conditioned Gating for Robust CT Triage Under Distribution Shift](https://mlsanity.com/paper/2605.13813/) (2026)

## Abstract (arXiv)

Chest CT is among the highest-volume imaging exams in medicine, yet expert voxel-level annotations are scarce and costly, motivating encoders that learn directly from unlabeled scans. We present DALE-CT, a family of 2D slice-based Vision Transformers trained from scratch on chest CT with the heuristics-free LeJEPA objective. We introduce depth-aware slab sampling, which draws self-supervised views from across a physical z-axis slab rather than a single slice, implicitly tasking the 2D encoder with representing how anatomy changes between neighboring slices. The frozen representations trace each scan as a smooth anatomical trajectory, recover cranio-caudal slice ordering without labels, and distinguish slices by the anatomy they contain rather than by position alone. This anatomical world model emerges without any 3D or positional supervision, and an otherwise-identical encoder trained on slices in isolation never develops it. Building on this backbone, we introduce dense auxiliary supervision into the pretraining objective, using anatomical and abnormality masks to supervise patch and slice tokens alongside the self-supervised loss, and we compare the resulting variants against a DINOv2 baseline continually pretrained on CT-RATE. Among nine public and in-house models evaluated under the same protocol, DALE-CT-2S is the strongest 2D model in-domain, reaching 0.825 Macro AUROC on CT-RATE, within 0.024 of COLIPRI-CRM and without any text supervision. We subsequently scale the supervision-free configuration to a ~287k-scan multi-source pool, to our knowledge the largest reported chest-CT pretraining corpus. The resulting DALE-CT-0-L posts the best 2D external-transfer point estimates, and we release it as our recommended backbone with the full model family, training code, and evaluation pipeline.