mlsanityData to 14 Oct 2026

Flow as the Cross-Domain Manipulation Interface

arXiv 2407.15208 ↗First posted 21 Jul 2024cs.RO cs.AIcited by 132

Not on the map. Why?

Cited by 25 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Im2Flow2Act, a scalable learning framework that enables robots to acquire real-world manipulation skills without the need of real-world robot training data. The key idea behind Im2Flow2Act is to use object flow as the manipulation interface, bridging domain gaps between different embodiments (i.e., human and robot) and training environments (i.e., real-world and simulated). Im2Flow2Act comprises two components: a flow generation network and a flow-conditioned policy. The flow generation network, trained on human demonstration videos, generates object flow from the initial scene image, conditioned on the task description. The flow-conditioned policy, trained on simulated robot play data, maps the generated object flow to robot actions to realize the desired object movements. By using flow as input, this policy can be directly deployed in the real world with a minimal sim-to-real gap. By leveraging real-world human videos and simulated robot play data, we bypass the challenges of teleoperating physical robots in the real world, resulting in a scalable system for diverse tasks. We demonstrate Im2Flow2Act's capabilities in a variety of real-world tasks, including the manipulation of rigid, articulated, and deformable objects.

Key facts. Cited by 132 papers in the map (data through 14 Oct 2026). Builds on 58 of its 68 references that are in the map. First posted on arXiv 21 Jul 2024.

Builds on

58 of 68 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  10. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
38 more
  1. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  2. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  3. 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 294
  4. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 287
  5. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  6. 2022Masked Visual Pre-training for Motor Controlcited by 248
  7. 2019Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Datacited by 246
  8. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  9. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
  10. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  11. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  12. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  13. 2019Implicit 3D Orientation Learning for 6D Object Detection from RGB Imagescited by 170
  14. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  15. 2022Human-to-Robot Imitation in the Wildcited by 161
  16. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 160
  17. 2023Structured World Models from Human Videoscited by 147
  18. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 136
  19. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  20. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
  21. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  22. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
  23. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  24. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  25. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 90
  26. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  27. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  28. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 84
  29. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 82
  30. 2023XSkill: Cross Embodiment Skill Discoverycited by 75
  31. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  32. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 72
  33. 2022ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Cloudscited by 54
  34. 2023Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperationcited by 54
  35. 2021Human Hands as Probes for Interactive Object Understandingcited by 52
  36. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
  37. 2020Sim2Real Transfer for Reinforcement Learning without Dynamics Randomizationcited by 47
  38. 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 61; 2026: 63.

132 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  2. 2025Unified Video Action Modelcited by 205
  3. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  4. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  5. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  7. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 59
  8. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 56
  9. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 51
  10. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  11. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  12. 2025H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulationcited by 45
38 more
  1. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 43
  2. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 42
  3. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
  4. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 40
  5. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
  6. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  7. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 37
  8. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 37
  9. 2026World Guidance: World Modeling in Condition Space for Action Generationcited by 36
  10. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 35
  11. 2025UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representationscited by 35
  12. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 32
  13. 2026CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videoscited by 31
  14. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 29
  15. 2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 29
  16. 2025Pre-training Auto-regressive Robotic Models with 4D Representationscited by 27
  17. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  18. 2025TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulationcited by 24
  19. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 24
  20. 2025Zero-Shot Monocular Scene Flow Estimation in the Wildcited by 22
  21. 2025Robotic Visual Instructioncited by 22
  22. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
  23. 2025Robot Learning from a Physical World Modelcited by 20
  24. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 20
  25. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 18
  26. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  27. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  28. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 16
  29. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 16
  30. 2026MaskWAM: Unifying Mask Prompting and Prediction for World-Action Modelscited by 16
  31. 2026FlowWAM: Optical Flow as a Unified Action Representation for World Action Modelscited by 16
  32. 2025Diffusion Trajectory-guided Policy for Long-horizon Robot Manipulationcited by 15
  33. 2025FlowMP: Learning Motion Fields for Robot Planning with Conditional Flow Matchingcited by 15
  34. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  35. 2025TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videoscited by 15
  36. 2025Action Tokenizer Matters in In-Context Imitation Learningcited by 14
  37. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 14
  38. 2026Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulationcited by 14

50 most cited of 132 All 132 (JSON)