# DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers

arXiv: https://arxiv.org/abs/2503.14405 · HTML: https://mlsanity.com/paper/2503.14405/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 65 of its 75 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval](https://mlsanity.com/paper/2004.01804/) (2020)

## Built on by (most cited first, 30 of 32)

- [HAMSt3R: Human-Aware Multi-view Stereo 3D Reconstruction](https://mlsanity.com/paper/2508.16433/) (2025)
- [UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding](https://mlsanity.com/paper/2508.11952/) (2025)
- [Chorus: Multi-Teacher Pretraining for Holistic 3D Gaussian Scene Encoding](https://mlsanity.com/paper/2512.17817/) (2025)
- [Efficient Universal Perception Encoder](https://mlsanity.com/paper/2603.22387/) (2026)
- [TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learning](https://mlsanity.com/paper/2512.21331/) (2025)
- [ProgVLA: Progress-Aware Robot Manipulation Skill Learning](https://mlsanity.com/paper/2605.28231/) (2026)
- [Distill3R: A Pipeline for Democratizing 3D Foundation Models on Commodity Hardware](https://mlsanity.com/paper/2602.00865/) (2026)
- [Brewing Stronger Features: Dual-Teacher Distillation for Multispectral Earth Observation](https://mlsanity.com/paper/2602.19863/) (2026)
- [CAM3R: Camera-Agnostic Model for 3D Reconstruction](https://mlsanity.com/paper/2603.22631/) (2026)
- [Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillation](https://mlsanity.com/paper/2602.06032/) (2026)
- [Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction](https://mlsanity.com/paper/2604.11707/) (2026)
- [Compressing History into Memory: Distilling Transformers into Recurrent Transformers](https://mlsanity.com/paper/2606.21562/) (2026)
- [TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation](https://mlsanity.com/paper/2607.10762/) (2026)
- [Vanishing Depth: Training Generalized Depth Adapters with Sinusoidal Depth Preprocessing for Pretrained RGB Encoders](https://mlsanity.com/paper/2503.19947/) (2025)
- [Rank-Aware Agglomeration of Foundation Models for Immunohistochemistry Image Cell Counting](https://mlsanity.com/paper/2511.12588/) (2025)
- [SPIDER: Spatial Image CorresponDence Estimator for Robust Calibration](https://mlsanity.com/paper/2511.17750/) (2025)
- [InCaRPose: In-Cabin Relative Camera Pose Estimation Model and Dataset](https://mlsanity.com/paper/2604.03814/) (2026)
- [Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks](https://mlsanity.com/paper/2604.12935/) (2026)
- [Multimodal embodiment-aware navigation transformer](https://mlsanity.com/paper/2604.19267/) (2026)
- [REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception](https://mlsanity.com/paper/2605.00271/) (2026)
- [SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models](https://mlsanity.com/paper/2605.31597/) (2026)
- [PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization](https://mlsanity.com/paper/2606.03444/) (2026)
- [Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstruction](https://mlsanity.com/paper/2607.01851/) (2026)
- [SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignment](https://mlsanity.com/paper/2607.15058/) (2026)
- [DDMS: Discriminative Distillation of Multi-view Foundational Features into Single-view Models](https://mlsanity.com/paper/2608.23850/) (2026)
- [IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves](https://mlsanity.com/paper/2608.24759/) (2026)
- [DINOcular: Self-Supervised Visuospatial Representations](https://mlsanity.com/paper/2608.27226/) (2026)
- [OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute](https://mlsanity.com/paper/2609.36374/) (2026)
- [GRAFT: Growing Agglomerative Foundation Models via Continual Teacher Distillation](https://mlsanity.com/paper/2610.02597/) (2026)
- [Scaling Multi-Teacher Distillation for Digital Pathology](https://mlsanity.com/paper/2610.12574/) (2026)

## Abstract (arXiv)

Recent multi-teacher distillation methods have unified the encoders of multiple foundation models into a single encoder, achieving competitive performance on core vision tasks like classification, segmentation, and depth estimation. This led us to ask: Could similar success be achieved when the pool of teachers also includes vision models specialized in diverse tasks across both 2D and 3D perception? In this paper, we define and investigate the problem of heterogeneous teacher distillation, or co-distillation, a challenging multi-teacher distillation scenario where teacher models vary significantly in both (a) their design objectives and (b) the data they were trained on. We explore data-sharing strategies and teacher-specific encoding, and introduce DUNE, a single encoder excelling in 2D vision, 3D understanding, and 3D human perception. Our model achieves performance comparable to that of its larger teachers, sometimes even outperforming them, on their respective tasks. Notably, DUNE surpasses MASt3R in Map-free Visual Relocalization with a much smaller encoder.