# Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

arXiv: https://arxiv.org/abs/2609.38172 · HTML: https://mlsanity.com/paper/2609.38172/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 29 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [DreamGen: Unlocking Generalization in Robot Learning through Video World Models](https://mlsanity.com/paper/2505.12705/) (2025)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [Seedance 2.0: Advancing Video Generation for World Complexity](https://mlsanity.com/paper/2604.14148/) (2026)
- [Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2409.16283/) (2024)
- [OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction](https://mlsanity.com/paper/2509.26633/) (2025)
- [Large Video Planner Enables Generalizable Robot Control](https://mlsanity.com/paper/2512.15840/) (2025)
- [Visual Imitation Enables Contextual Humanoid Control](https://mlsanity.com/paper/2505.03729/) (2025)
- [Human-Object Interaction from Human-Level Instructions](https://mlsanity.com/paper/2406.17840/) (2024)
- [Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations](https://mlsanity.com/paper/2507.00990/) (2025)
- [HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos](https://mlsanity.com/paper/2509.16757/) (2025)
- [Learning from Massive Human Videos for Universal Humanoid Pose Control](https://mlsanity.com/paper/2412.14172/) (2024)
- [Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching](https://mlsanity.com/paper/2512.11130/) (2025)
- [MonoFusion: Sparse-View 4D Reconstruction via Monocular Fusion](https://mlsanity.com/paper/2507.23782/) (2025)
- [Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching](https://mlsanity.com/paper/2602.15827/) (2026)
- [HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos](https://mlsanity.com/paper/2602.02473/) (2026)
- [Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation](https://mlsanity.com/paper/2602.15828/) (2026)
- [ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation](https://mlsanity.com/paper/2603.03279/) (2026)
- [LessMimic: Long-Horizon Humanoid Interaction with Unified Distance Field Representations](https://mlsanity.com/paper/2602.21723/) (2026)
- [VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes](https://mlsanity.com/paper/2606.30645/) (2026)
- [Generalizing from References using a Multi-Task Reference and Goal-Driven RL Framework](https://mlsanity.com/paper/2602.20375/) (2026)

## Abstract (arXiv)

Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical barrier to scaling this approach. We propose PRISM, a real-to-sim-to-real framework that overcomes this limitation by amplifying a handful of real videos into a large, diverse training set. PRISM first generates hundreds of diverse "counterfactual" human-object interaction videos via video-to-video (V2V) generation from a few exemplar real videos. Our contact-anchored real-to-sim pipeline then reconstructs both human and object motions, retargeting this imperfect video data into physically plausible trajectories. The intra-class variability across these counterfactual videos lets us train a single policy that generalizes to unseen objects within each category. We demonstrate the full pipeline by deploying this policy on a real robot without any real-world fine-tuning. Using only onboard depth observations, our humanoid picks up, carries, and drops objects, including boxes, barrels, bins, and balls, across novel instances, sizes, and initial configurations.