mlsanityData to 14 Oct 2026

Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation

arXiv 2412.15109 ↗First posted 19 Dec 2024cs.ROcited by 161

On the map: World models → Embodied and robot world models

Abstract (arXiv)

Current efforts to learn scalable policies in robotic manipulation primarily fall into two categories: one focuses on "action," which involves behavior cloning from extensive collections of robotic data, while the other emphasizes "vision," enhancing model generalization by pre-training representations or generative models, also referred to as world models, using large-scale visual datasets. This paper presents an end-to-end paradigm that predicts actions using inverse dynamics models conditioned on the robot's forecasted visual states, named Predictive Inverse Dynamics Models (PIDM). By closing the loop between vision and action, the end-to-end PIDM can be a better scalable action learner. In practice, we use Transformers to process both visual states and actions, naming the model Seer. It is initially pre-trained on large-scale robotic datasets, such as DROID, and can be adapted to realworld scenarios with a little fine-tuning data. Thanks to large-scale, end-to-end training and the synergy between vision and action, Seer significantly outperforms previous methods across both simulation and real-world experiments. It achieves improvements of 13% on the LIBERO-LONG benchmark, 21% on CALVIN ABC-D, and 43% in real-world tasks. Notably, Seer sets a new state-of-the-art on CALVIN ABC-D benchmark, achieving an average length of 4.28, and exhibits superior generalization for novel objects, lighting conditions, and environments under high-intensity disturbances on real-world scenarios. Code and models are publicly available at https://github.com/OpenRobotLab/Seer/.

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 38 of its 40 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

38 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI17 Oct
  2. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  4. 2022Masked World Models for Visual Control28 Jun
  5. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  6. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physics23 Jun

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  8. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  10. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  11. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
20 more
  1. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  3. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  4. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  5. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  6. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 349
  7. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  8. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  9. 2022Masked Visual Pre-training for Motor Controlcited by 248
  10. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  11. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  12. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
  13. 2023Language-Driven Representation Learning for Roboticscited by 165
  14. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  15. 2024Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulationcited by 53
  16. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  17. 2024Learning Manipulation by Predicting Interactioncited by 37
  18. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 31
  19. 2024VideoAgent: Self-Improving Video Generationcited by 25
  20. 2023Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Trainingcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 70; 2026: 91.

161 papers.

Sort:

By area:

  1. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  2. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  3. 2026Causal World Modeling for Robot Controlcited by 321
  4. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  5. 2025Motus: A Unified Latent Action World Modelcited by 251
  6. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  7. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 145
  8. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 135
  9. 2025Unified Vision-Language-Action Modelcited by 134
  10. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 123
  11. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  12. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
38 more
  1. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 86
  2. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 84
  3. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 81
  4. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 76
  5. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 74
  6. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 72
  7. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 61
  8. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 57
  9. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 51
  10. 2025InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policycited by 50
  11. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 50
  12. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
  13. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 47
  14. 2025DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoningcited by 46
  15. 2026AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routingcited by 46
  16. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 45
  17. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 41
  18. 2025Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Processcited by 39
  19. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 38
  20. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  21. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 36
  22. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
  23. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  24. 2025HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Modelscited by 32
  25. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 31
  26. 2026LaST0: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Modelcited by 31
  27. 2026Self-Supervised Flow Matching for Scalable Multi-Modal Synthesiscited by 31
  28. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  29. 2026ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queriescited by 29
  30. 2025From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priorscited by 28
  31. 2025CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modelingcited by 25
  32. 2025RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulationcited by 25
  33. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 25
  34. 2025LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulationcited by 24
  35. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 24
  36. 2025AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulationcited by 23
  37. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulationcited by 23
  38. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 22

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
  2. 2026World Models for Robotic Manipulation: A Survey
  3. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrations
  4. 2026ZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training
  5. 2026Learning Transferable Dynamics Priors from Action to World Modeling
  6. 2026InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
  7. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs
  8. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation