mlsanityData to 7 Oct 2026

General Flow as Foundation Affordance for Scalable Robot Learning

arXiv 2401.11439 ↗First posted 21 Jan 2024cs.RO cs.AI cs.CVcited by 80

Not on the map. Why?

Abstract (arXiv)

We address the challenge of acquiring real-world manipulation skills with a scalable framework. We hold the belief that identifying an appropriate prediction target capable of leveraging large-scale datasets is crucial for achieving efficient and universal learning. Therefore, we propose to utilize 3D flow, which represents the future trajectories of 3D points on objects of interest, as an ideal prediction target. To exploit scalable data resources, we turn our attention to human videos. We develop, for the first time, a language-conditioned 3D flow prediction model directly from large-scale RGBD human video datasets. Our predicted flow offers actionable guidance, thus facilitating zero-shot skill transfer in real-world scenarios. We deploy our method with a policy based on closed-loop flow prediction. Remarkably, without any in-domain finetuning, our method achieves an impressive 81% success rate in zero-shot human-to-robot skill transfer, covering 18 tasks in 6 scenes. Our framework features the following benefits: (1) scalability: leveraging cross-embodiment data resources; (2) wide application: multiple object categories, including rigid, articulated, and soft bodies; (3) stable skill transfer: providing actionable guidance with a small inference domain-gap. Code, data, and supplementary materials are available https://general-flow.github.io

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 89 of its 106 references that are in the map. First posted on arXiv 21 Jan 2024.

Builds on

89 of 106 references.

References · 89

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  6. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  9. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  10. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  11. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  12. 2023CoTracker: It is Better to Track Togethercited by 516
  13. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  14. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  15. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  16. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  17. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  18. 2023Fast Segment Anythingcited by 286
  19. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  20. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  21. 2022Masked Visual Pre-training for Motor Controlcited by 246
  22. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  23. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  24. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  25. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  26. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  27. 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 205
  28. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  29. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  30. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  31. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  32. 2023Video Language Planningcited by 160
  33. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  34. 2022Human-to-Robot Imitation in the Wildcited by 157
  35. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  36. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  37. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  38. 2023Structured World Models from Human Videoscited by 144

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 40; 2026: 30.

80 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  3. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  4. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  5. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 115
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  7. 2025TAPIP3D: Tracking Any Point in Persistent 3D Geometrycited by 67
  8. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  9. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  10. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  11. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  12. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
38 more
  1. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  2. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 38
  3. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  4. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36
  5. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  6. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 35
  7. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  8. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  9. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  10. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 27
  11. 2026CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videoscited by 27
  12. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
  13. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 21
  14. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  15. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  16. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 14
  17. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 13
  18. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
  19. 2025EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flowcited by 10
  20. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  21. 2025GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulationcited by 9
  22. 20253D Flow Diffusion Policy: Visuomotor Policy Learning via Generating Flow in 3D Spacecited by 9
  23. 2025METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Modelcited by 9
  24. 2025Latent Policy Steering with Embodiment-Agnostic Pretrained World Modelscited by 8
  25. 2025GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learningcited by 8
  26. 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understandingcited by 8
  27. 2025Pixel Motion Diffusion is What We Need for Robot Controlcited by 7
  28. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  29. 2024EqvAfford: SE(3) Equivariance for Point-Level Affordance Learningcited by 6
  30. 2025Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videoscited by 6
  31. 2025ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flowcited by 5
  32. 2026GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planningcited by 5
  33. 2025Self-Supervised Learning of Motion Concepts by Optimizing Counterfactualscited by 4
  34. 2025Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generationcited by 4
  35. 2025See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrationscited by 4
  36. 2026Forecasting Motion in the Wildcited by 4
  37. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  38. 2026πR2: Reactive Real-time Flow Policiescited by 4

50 most cited of 80 All 80 (JSON)