# 3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model

arXiv: https://arxiv.org/abs/2506.06199 · HTML: https://mlsanity.com/paper/2506.06199/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models](https://mlsanity.com/paper/2402.07865/) (2024)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [Any-point Trajectory Modeling for Policy Learning](https://mlsanity.com/paper/2401.00025/) (2023)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)
- [RoboDreamer: Learning Compositional World Models for Robot Imagination](https://mlsanity.com/paper/2404.12377/) (2024)
- [RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot](https://mlsanity.com/paper/2307.00595/) (2023)
- [Scaling Robot Learning with Semantically Imagined Experience](https://mlsanity.com/paper/2302.11550/) (2023)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2409.16283/) (2024)
- [iVideoGPT: Interactive VideoGPTs are Scalable World Models](https://mlsanity.com/paper/2405.15223/) (2024)

## Built on by (most cited first, 30 of 37)

- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [World-in-World: World Models in a Closed-Loop World](https://mlsanity.com/paper/2510.18135/) (2025)
- [NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos](https://mlsanity.com/paper/2510.08568/) (2025)
- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow](https://mlsanity.com/paper/2512.24766/) (2025)
- [A Step Toward World Models: A Survey on Robotic Manipulation](https://mlsanity.com/paper/2511.02097/) (2025)
- [TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videos](https://mlsanity.com/paper/2511.21690/) (2025)
- [Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation](https://mlsanity.com/paper/2602.15828/) (2026)
- [Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test](https://mlsanity.com/paper/2601.04137/) (2026)
- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [LaMP: Learning Vision-Language-Action Policy with 3D Scene Flow as Latent Motion Prior](https://mlsanity.com/paper/2603.25399/) (2026)
- [GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization](https://mlsanity.com/paper/2605.12369/) (2026)
- [3D Flow Diffusion Policy: Visuomotor Policy Learning via Generating Flow in 3D Space](https://mlsanity.com/paper/2509.18676/) (2025)
- [Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI](https://mlsanity.com/paper/2606.16533/) (2026)
- [GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation](https://mlsanity.com/paper/2605.22882/) (2026)
- [Cross-Hand Latent Representation for Vision-Language-Action Models](https://mlsanity.com/paper/2603.10158/) (2026)
- [RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation](https://mlsanity.com/paper/2605.17522/) (2026)
- [Track4Action: Distilling World-Centric 3D Tracker into Vision-Language-Action Policies](https://mlsanity.com/paper/2608.03727/) (2026)
- [From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Models](https://mlsanity.com/paper/2601.15533/) (2026)
- [From World Models to World Action Models: A Concise Tutorial for Robotics](https://mlsanity.com/paper/2607.00836/) (2026)
- [SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space](https://mlsanity.com/paper/2608.01397/) (2026)
- [Foresight Without Seeing: Latent Futures for World Action Models](https://mlsanity.com/paper/2608.11605/) (2026)
- [μ0: A Scalable 3D Interaction-Trace World Model](https://mlsanity.com/paper/2606.13769/) (2026)
- [Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning](https://mlsanity.com/paper/2512.05953/) (2025)
- [Embodied Tree of Thoughts: Deliberate Manipulation Planning with Embodied World Model](https://mlsanity.com/paper/2512.08188/) (2025)
- [Overcoming Dynamics-Blindness: Training-Free Pace-and-Path Correction for VLA Models](https://mlsanity.com/paper/2605.11459/) (2026)
- [Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations](https://mlsanity.com/paper/2605.22123/) (2026)
- [Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer](https://mlsanity.com/paper/2607.28394/) (2026)
- [The Embodiment Gap in Robot Foundation Models](https://mlsanity.com/paper/2608.18433/) (2026)

## Abstract (arXiv)

Manipulation has long been a challenging task for robots, while humans can effortlessly perform complex interactions with objects, such as hanging a cup on the mug rack. A key reason is the lack of a large and uniform dataset for teaching robots manipulation skills. Current robot datasets often record robot action in different action spaces within a simple scene. This hinders the robot to learn a unified and robust action representation for different robots within diverse scenes. Observing how humans understand a manipulation task, we find that understanding how the objects should move in the 3D space is a critical clue for guiding actions. This clue is embodiment-agnostic and suitable for both humans and different robots. Motivated by this, we aim to learn a 3D flow world model from both human and robot manipulation data. This model predicts the future movement of the interacting objects in 3D space, guiding action planning for manipulation. Specifically, we synthesize a large-scale 3D optical flow dataset, named ManiFlow-110k, through a moving object auto-detect pipeline. A video diffusion-based world model then learns manipulation physics from these data, generating 3D optical flow trajectories conditioned on language instructions. With the generated 3D object optical flow, we propose a flow-guided rendering mechanism, which renders the predicted final state and leverages GPT-4o to assess whether the predicted flow aligns with the task description. This equips the robot with a closed-loop planning ability. Finally, we consider the predicted 3D optical flow as constraints for an optimization policy to determine a chunk of robot actions for manipulation. Extensive experiments demonstrate strong generalization across diverse robotic manipulation tasks and reliable cross-embodiment adaptation without hardware-specific training.