mlsanityData to 14 Oct 2026

DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers

arXiv 2503.14405 ↗First posted 18 Mar 2025cs.CV cs.LGcited by 32

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

Recent multi-teacher distillation methods have unified the encoders of multiple foundation models into a single encoder, achieving competitive performance on core vision tasks like classification, segmentation, and depth estimation. This led us to ask: Could similar success be achieved when the pool of teachers also includes vision models specialized in diverse tasks across both 2D and 3D perception? In this paper, we define and investigate the problem of heterogeneous teacher distillation, or co-distillation, a challenging multi-teacher distillation scenario where teacher models vary significantly in both (a) their design objectives and (b) the data they were trained on. We explore data-sharing strategies and teacher-specific encoding, and introduce DUNE, a single encoder excelling in 2D vision, 3D understanding, and 3D human perception. Our model achieves performance comparable to that of its larger teachers, sometimes even outperforming them, on their respective tasks. Notably, DUNE surpasses MASt3R in Map-free Visual Relocalization with a much smaller encoder.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 65 of its 75 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

65 of 75 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  10. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  12. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
38 more
  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  2. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  3. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 984
  4. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
  5. 2023Vision Transformers Need Registerscited by 871
  6. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  7. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  8. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  9. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  10. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  11. 2020AdaBins: Depth Estimation using Adaptive Binscited by 601
  12. 2020FDA: Fourier Domain Adaptation for Semantic Segmentationcited by 541
  13. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 537
  14. 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 506
  15. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 490
  16. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  17. 2020Virtual KITTI 2cited by 353
  18. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  19. 2023BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motioncited by 236
  20. 2021AGORA: Avatars in Geography Optimized for Regression Analysiscited by 205
  21. 2018DeepV2D: Video to Depth with Differentiable Structure from Motioncited by 185
  22. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  23. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 173
  24. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  25. 2023PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustmentcited by 147
  26. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  27. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  28. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 130
  29. 2021SMD-Nets: Stereo Mixture Density Networkscited by 120
  30. 2024RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videoscited by 115
  31. 2022Map-free Visual Relocalization: Metric Pose Relative to a Single Imagecited by 114
  32. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  33. 2024Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shotcited by 101
  34. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  35. 2024Model Stock: All we need is just a few fine-tuned modelscited by 92
  36. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  37. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  38. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 69

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 25.

32 papers.

Sort:

By area:

  1. 2025HAMSt3R: Human-Aware Multi-view Stereo 3D Reconstructioncited by 15
  2. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 10
  3. 2025Chorus: Multi-Teacher Pretraining for Holistic 3D Gaussian Scene Encodingcited by 10
  4. 2026Efficient Universal Perception Encodercited by 10
  5. 2025TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learningcited by 6
  6. 2026ProgVLA: Progress-Aware Robot Manipulation Skill Learningcited by 4
  7. 2026Distill3R: A Pipeline for Democratizing 3D Foundation Models on Commodity Hardwarecited by 3
  8. 2026Brewing Stronger Features: Dual-Teacher Distillation for Multispectral Earth Observationcited by 2
  9. 2026CAM3R: Camera-Agnostic Model for 3D Reconstructioncited by 2
  10. 2026Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillationcited by 1
  11. 2026Representations Before Pixels: Semantics-Guided Hierarchical Video Predictioncited by 1
  12. 2026Compressing History into Memory: Distilling Transformers into Recurrent Transformerscited by 1
20 more
  1. 2026TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillationcited by 1
  2. 2025Vanishing Depth: Training Generalized Depth Adapters with Sinusoidal Depth Preprocessing for Pretrained RGB Encoderscited by 0
  3. 2025Rank-Aware Agglomeration of Foundation Models for Immunohistochemistry Image Cell Countingcited by 0
  4. 2025SPIDER: Spatial Image CorresponDence Estimator for Robust Calibrationcited by 0
  5. 2026InCaRPose: In-Cabin Relative Camera Pose Estimation Model and Datasetcited by 0
  6. 2026Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Taskscited by 0
  7. 2026Multimodal embodiment-aware navigation transformercited by 0
  8. 2026REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perceptioncited by 0
  9. 2026SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Modelscited by 0
  10. 2026PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specializationcited by 0
  11. 2026Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstructioncited by 0
  12. 2026SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignmentcited by 0
  13. 2026DDMS: Discriminative Distillation of Multi-view Foundational Features into Single-view Modelscited by 0
  14. 2026IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leavescited by 0
  15. 2026DINOcular: Self-Supervised Visuospatial Representationscited by 0
  16. 2026OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Computecited by 0
  17. 2026GRAFT: Growing Agglomerative Foundation Models via Continual Teacher Distillationcited by 0
  18. 2026Scaling Multi-Teacher Distillation for Digital Pathologycited by 0
  19. 2026Recovering from Pose-Selection Failures in Repeated Visual Localizationcited by 0
  20. 2026ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models
  2. 2024Multi-View 3D Reconstruction using Knowledge Distillation
  3. 2025Video Self-Distillation for Single-Image Encoders: A Step Toward Physically Plausible Perception
  4. 2026The Professor: Multi-Teacher Unsupervised Prompt Distillation for Vision-Language Models
  5. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Models
  6. 2024TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation
  7. 2023ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation
  8. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation