mlsanityData to 7 Oct 2026

Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers

arXiv 2409.20537 ↗First posted 30 Sep 2024cs.RO cs.CV cs.LGcited by 145

Not on the map. Why?

Cited by 71 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One of the roadblocks for training generalist robotic models today is heterogeneity. Previous robot learning methods often collect data to train with one specific embodiment for one task, which is expensive and prone to overfitting. This work studies the problem of learning policy representations through heterogeneous pre-training on robot data across different embodiments and tasks at scale. We propose Heterogeneous Pre-trained Transformers (HPT), which pre-train a large, shareable trunk of a policy neural network to learn a task and embodiment agnostic shared representation. This general architecture aligns the specific proprioception and vision inputs from distinct embodiments to a short sequence of tokens and then processes such tokens to map to control robots for different tasks. Leveraging the recent large-scale multi-embodiment real-world robotic datasets as well as simulation, deployed robots, and human video datasets, we investigate pre-training policies across heterogeneity. We conduct experiments to investigate the scaling behaviors of training objectives, to the extent of 52 datasets. HPTs outperform several baselines and enhance the fine-tuned policy performance by over 20% on unseen tasks in multiple simulator benchmarks and real-world settings. See the project website (https://liruiw.github.io/hpt/) for code and videos.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 69 of its 89 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

69 of 89 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  13. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  14. 2024Mixtral of Expertscited by 1,785
  15. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  16. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  17. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  18. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  19. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  20. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  21. 2022A Generalist Agentcited by 802
  22. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  23. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  24. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  25. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  26. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  27. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  28. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  29. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  30. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  31. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  32. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  33. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  34. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  35. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  36. 2022Masked World Models for Visual Controlcited by 169
  37. 2023Language-Driven Representation Learning for Roboticscited by 163
  38. 2024Behavior Generation with Latent Actionscited by 159

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 72; 2026: 65.

145 papers.

Sort:

By area:

  1. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  2. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  3. 2025Motus: A Unified Latent Action World Modelcited by 231
  4. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  5. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  6. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  7. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  8. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  9. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  10. 2025GR-3 Technical Reportcited by 110
  11. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  12. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
38 more
  1. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  2. 2025Humanoid Policy ~ Human Policycited by 65
  3. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 61
  4. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  5. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  6. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  7. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 55
  8. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  9. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 48
  10. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  11. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  12. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  13. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  14. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39
  15. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  16. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  17. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  18. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  19. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 29
  20. 2025EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Datacited by 27
  21. 2026CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videoscited by 27
  22. 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systemscited by 27
  23. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  24. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 25
  25. 2025Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulationcited by 24
  26. 2025Embodied AI: From LLMs to World Modelscited by 23
  27. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
  28. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 21
  29. 2025Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimizationcited by 20
  30. 2025Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspectivecited by 20
  31. 2026HoloBrain-0 Technical Reportcited by 20
  32. 2025Is Diversity All You Need for Scalable Robotic Manipulation?cited by 19
  33. 2025FedVLA: Federated Vision-Language-Action Learning with Dual Gating Mixture-of-Experts for Robotic Manipulationcited by 18
  34. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  35. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  36. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  37. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 17
  38. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 17

50 most cited of 145 All 145 (JSON)