mlsanityData to 7 Oct 2026

Learning to Act from Actionless Videos through Dense Correspondences

arXiv 2310.08576 ↗First posted 12 Oct 2023cs.RO cs.CV cs.LGcited by 159

Not on the map. Why?

Cited by 54 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments from few video demonstrations without using any action annotations. Our method leverages images as a task-agnostic representation, encoding both the state and action information, and text as a general representation for specifying robot goals. By synthesizing videos that ``hallucinate'' robot executing actions and in combination with dense correspondences between frames, our approach can infer the closed-formed action to execute to an environment without the need of any explicit action labels. This unique capability allows us to train the policy solely based on RGB videos and deploy learned policies to various robotic tasks. We demonstrate the efficacy of our approach in learning policies on table-top manipulation and navigation tasks. Additionally, we contribute an open-source framework for efficient video modeling, enabling the training of high-fidelity policy models with four GPUs within a single day.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

47 of 55 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  8. 2022Video Diffusion Modelscited by 2,280
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
35 more
  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  2. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  3. 2018Behavioral Cloning from Observationcited by 482
  4. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  5. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  6. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  7. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  8. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  9. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  10. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  11. 2018Generative Adversarial Imitation from Observationcited by 183
  12. 2023Language-Driven Representation Learning for Roboticscited by 163
  13. 2019Goal-conditioned Imitation Learningcited by 159
  14. 2022Human-to-Robot Imitation in the Wildcited by 157
  15. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  16. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  17. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  18. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  19. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  20. 2019Recent Advances in Imitation Learning from Observationcited by 92
  21. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  22. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  23. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  24. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 76
  25. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  26. 2018Learning Plannable Representations with Causal InfoGANcited by 56
  27. 2022Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learningcited by 54
  28. 2020Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Datacited by 51
  29. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  30. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  31. 2015Human Action Recognition using Factorized Spatio-Temporal Convolutional Networkscited by 40
  32. 2023Diffusion Model-Augmented Behavioral Cloningcited by 37
  33. 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Trainingcited by 31
  34. 2023Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulationcited by 21
  35. 2022Adversarial Imitation Learning from Video using a State Observercited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 38; 2025: 70; 2026: 48.

159 papers.

Sort:

By area:

  1. 2026World Action Models are Zero-shot Policiescited by 411
  2. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  4. 2024Latent Action Pretraining from Videoscited by 302
  5. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  6. 2025Unified Video Action Modelcited by 195
  7. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  8. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  9. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  10. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  11. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  12. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
38 more
  1. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  2. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  3. 2025Controllable Video Generation: A Surveycited by 93
  4. 2024Video as the New Language for Real-World Decision Makingcited by 86
  5. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  7. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  8. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  9. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  10. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  11. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  12. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  13. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 61
  14. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  15. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  16. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  17. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  18. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  19. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  20. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  21. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  22. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  23. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  24. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 42
  25. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  26. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  27. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  28. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39
  29. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
  30. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 38
  31. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 37
  32. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36
  33. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  34. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  35. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  36. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  37. 2025EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modelingcited by 29
  38. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28

50 most cited of 159 All 159 (JSON)