mlsanityData to 7 Oct 2026

Theia: Distilling Diverse Vision Foundation Models for Robot Learning

arXiv 2407.20179 ↗First posted 29 Jul 2024cs.RO cs.AI cs.CVcited by 56

Not on the map. Why?

Abstract (arXiv)

Vision-based robot policy learning, which maps visual inputs to actions, necessitates a holistic understanding of diverse visual tasks beyond single-task needs like classification or segmentation. Inspired by this, we introduce Theia, a vision foundation model for robot learning that distills multiple off-the-shelf vision foundation models trained on varied vision tasks. Theia's rich visual representations encode diverse visual knowledge, enhancing downstream robot learning. Extensive experiments demonstrate that Theia outperforms its teacher models and prior robot learning models using less training data and smaller model sizes. Additionally, we quantify the quality of pre-trained visual representations and hypothesize that higher entropy in feature norm distributions leads to improved robot learning performance. Code, models, and demo are available at https://theia.theaiinstitute.com.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 68 of its 74 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

68 of 74 references.

References · 68

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2018DeepMind Control Suitecited by 935
  13. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  14. 2020Object-Centric Learning with Slot Attentioncited by 865
  15. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  16. 2023Vision Transformers Need Registerscited by 844
  17. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  18. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  19. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  20. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  21. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  22. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  23. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  24. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  25. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  26. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  27. 2020Reinforcement Learning with Augmented Datacited by 452
  28. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  29. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  30. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  31. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  32. 2022Masked Visual Pre-training for Motor Controlcited by 246
  33. 2023Track Anything: Segment Anything Meets Videoscited by 230
  34. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  35. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  36. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  37. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  38. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 27; 2026: 20.

56 papers.

Sort:

By area:

  1. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  3. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
  4. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  5. 2024SPA: 3D Spatial-Awareness Enables Effective Embodied Representationcited by 28
  6. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 28
  7. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 26
  8. 2024Feat2GS: Probing Visual Foundation Models with Gaussian Splattingcited by 26
  9. 2024Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasetscited by 23
  10. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representationscited by 21
  11. 2024PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillationcited by 17
  12. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
38 more
  1. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 12
  2. 2025GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulationcited by 11
  3. 2025AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learningcited by 8
  4. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  5. 2025Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-trainingcited by 7
  6. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 6
  7. 2025FeatSharp: Your Vision Model Features, Sharpercited by 6
  8. 2025RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Modelscited by 5
  9. 2025Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulationcited by 4
  10. 2025M3: 3D-Spatial MultiModal Memorycited by 3
  11. 2025Order Matters: On Parameter-Efficient Image-to-Video Probing for Recognizing Nearly Symmetric Actionscited by 3
  12. 2025Is an object-centric representation beneficial for robotic manipulation ?cited by 3
  13. 2025Model Parallelism With Subnetwork Data Parallelismcited by 3
  14. 2025VER: Vision Expert Transformer for Robot Learning via Foundation Distillation and Dynamic Routingcited by 3
  15. 2026DeFM: Learning Foundation Representations from Depth for Roboticscited by 3
  16. 2026When Does Predictive Inverse Dynamics Outperform Behavior Cloning?cited by 3
  17. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 2
  18. 2025Agglomerating Large Vision Encoders via Distillation for VFSS Segmentationcited by 2
  19. 2025Attentive Feature Aggregation or: How Policies Learn to Stop Worrying about Robustness and Attend to Task-Relevant Visual Cuescited by 2
  20. 2026X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learningcited by 2
  21. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Datacited by 1
  22. 2025Synthetic vs. Real Training Data for Visual Navigationcited by 1
  23. 2025SAM2Grasp: Resolve Multi-modal Grasping via Prompt-conditioned Temporal Action Predictioncited by 1
  24. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Modelscited by 1
  25. 2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
  26. 2026LIDEA: Human-to-Robot Imitation Learning via Implicit Feature Distillation and Explicit Geometry Alignmentcited by 1
  27. 2026GeoProp: Grounding Robot State in Vision for Generalist Manipulationcited by 1
  28. 2026TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillationcited by 1
  29. 2026Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistencycited by 1
  30. 2025Dynamic Pattern Alignment Learning for Pretraining Lightweight Human-Centric Vision Modelscited by 0
  31. 2025Exploring Conditions for Diffusion models in Robotic Controlcited by 0
  32. 2025Sceniris: A Fast Procedural Scene Generation Frameworkcited by 0
  33. 2026Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagationcited by 0
  34. 2026Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Corescited by 0
  35. 2026PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specializationcited by 0
  36. 2026Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generationcited by 0
  37. 2026Encoder Winners Do Not Reliably Transfer Across VLA Backbone Scale: A Frozen-Backbone Grafting Diagnosticcited by 0
  38. 2026UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learningcited by 0

50 most cited of 56 All 56 (JSON)