mlsanityData to 14 Oct 2026

Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation

arXiv 2409.16283 ↗First posted 24 Sep 2024cs.RO cs.CV cs.LGcited by 153

Not on the map. Why?

Cited by 55 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How can robot manipulation policies generalize to novel tasks involving unseen object types and new motions? In this paper, we provide a solution in terms of predicting motion information from web data through human video generation and conditioning a robot policy on the generated video. Instead of attempting to scale robot data collection which is expensive, we show how we can leverage video generation models trained on easily available web data, for enabling generalization. Our approach Gen2Act casts language-conditioned manipulation as zero-shot human video generation followed by execution with a single policy conditioned on the generated video. To train the policy, we use an order of magnitude less robot interaction data compared to what the video prediction model was trained on. Gen2Act doesn't require fine-tuning the video model at all and we directly use a pre-trained model for generating human videos. Our results on diverse real-world scenarios show how Gen2Act enables manipulating unseen object types and performing novel motions for tasks not present in the robot data. Videos are at https://homangab.github.io/gen2act/

Key facts. Cited by 153 papers in the map (data through 14 Oct 2026). Builds on 56 of its 62 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

56 of 62 references.

References · 56

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  9. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  10. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  12. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
38 more
  1. 2023CoTracker: It is Better to Track Togethercited by 522
  2. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  3. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  4. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  5. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  6. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 263
  7. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  8. 2022Masked Visual Pre-training for Motor Controlcited by 248
  9. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  10. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  11. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
  12. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  13. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  14. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  15. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 204
  16. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 182
  17. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  18. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 169
  19. 2023Language-Driven Representation Learning for Roboticscited by 165
  20. 2022Human-to-Robot Imitation in the Wildcited by 161
  21. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  22. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
  23. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 120
  24. 2020Visual Imitation Made Easycited by 111
  25. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
  26. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 106
  27. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
  28. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  29. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  30. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  31. 2024Video as the New Language for Real-World Decision Makingcited by 87
  32. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 87
  33. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  34. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 71
  35. 2023Dense Optical Tracking: Connecting the Dotscited by 61
  36. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 56
  37. 2021Human Hands as Probes for Interactive Object Understandingcited by 52
  38. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 60; 2026: 85.

153 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  2. 2026World Action Models are Zero-shot Policiescited by 465
  3. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  4. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  5. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 349
  6. 2024Latent Action Pretraining from Videoscited by 321
  7. 2026Causal World Modeling for Robot Controlcited by 321
  8. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  9. 2025Motus: A Unified Latent Action World Modelcited by 251
  10. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  11. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  12. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
38 more
  1. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 161
  2. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 123
  3. 2025Controllable Video Generation: A Surveycited by 98
  4. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  5. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 75
  6. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  7. 2025Large Video Planner Enables Generalizable Robot Controlcited by 70
  8. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 68
  9. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
  10. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 64
  11. 2025Vidar: Embodied Video Diffusion Model for Generalist Manipulationcited by 62
  12. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  13. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  14. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
  15. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 40
  16. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
  17. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 37
  18. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 37
  19. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
  20. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  21. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 32
  22. 2025Learning 3D Persistent Embodied World Modelscited by 29
  23. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 28
  24. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  25. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 24
  26. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 24
  27. 2025AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulationcited by 23
  28. 2026Light-WAM: Efficient World Action Models with State-Fusion Action Decodingcited by 23
  29. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 21
  30. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  31. 2025Robot Learning from a Physical World Modelcited by 20
  32. 2026RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulationcited by 19
  33. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  34. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 18
  35. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  36. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 18
  37. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  38. 2026MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulationcited by 17

50 most cited of 153 All 153 (JSON)