# ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection

arXiv: https://arxiv.org/abs/2405.03666 · HTML: https://mlsanity.com/paper/2405.03666/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 89 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [LIV: Language-Image Representations and Rewards for Robotic Control](https://mlsanity.com/paper/2306.00958/) (2023)
- [MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation](https://mlsanity.com/paper/2307.14460/) (2023)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [Structured World Models from Human Videos](https://mlsanity.com/paper/2308.10901/) (2023)
- [Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos](https://mlsanity.com/paper/2103.16817/) (2021)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [VideoDex: Learning Dexterity from Internet Videos](https://mlsanity.com/paper/2212.04498/) (2022)
- [DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video](https://mlsanity.com/paper/2202.00164/) (2022)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube](https://mlsanity.com/paper/2202.10448/) (2022)
- [FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding](https://mlsanity.com/paper/2105.03655/) (2021)
- [AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions](https://mlsanity.com/paper/2112.00246/) (2021)
- [FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy](https://mlsanity.com/paper/2111.05623/) (2021)
- [SpeedFolding: Learning Efficient Bimanual Folding of Garments](https://mlsanity.com/paper/2208.10552/) (2022)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)

## Built on by (most cited first, 19 of 19)

- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation](https://mlsanity.com/paper/2503.07135/) (2025)
- [R+X: Retrieval and Execution from Everyday Human Videos](https://mlsanity.com/paper/2407.12957/) (2024)
- [Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations](https://mlsanity.com/paper/2507.00990/) (2025)
- [ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos](https://mlsanity.com/paper/2503.23877/) (2025)
- [InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation](https://mlsanity.com/paper/2409.07914/) (2024)
- [VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulation](https://mlsanity.com/paper/2407.04152/) (2024)
- [ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation](https://mlsanity.com/paper/2602.00557/) (2026)
- [Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use](https://mlsanity.com/paper/2504.04612/) (2025)
- [SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation](https://mlsanity.com/paper/2506.15847/) (2025)
- [BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2406.10093/) (2024)
- [D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentation](https://mlsanity.com/paper/2505.04860/) (2025)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time](https://mlsanity.com/paper/2606.15631/) (2026)
- [Skills Made to Order: Efficient Acquisition of Robot Cooking Skills Guided by Multiple Forms of Internet Data](https://mlsanity.com/paper/2409.15172/) (2024)
- [Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress](https://mlsanity.com/paper/2509.24129/) (2025)
- [Robust Bimanual Vision-Language-Action Models via Embarrassingly Simple Modality Masking](https://mlsanity.com/paper/2608.22419/) (2026)
- [AsymDex: Asymmetry and Relative Coordinates for RL-based Bimanual Dexterity](https://mlsanity.com/paper/2411.13020/) (2024)
- [Learning Transferable World-Action Models from Task-Paired Human Videos](https://mlsanity.com/paper/2610.14351/) (2026)

## Abstract (arXiv)

Bimanual manipulation is a longstanding challenge in robotics due to the large number of degrees of freedom and the strict spatial and temporal synchronization required to generate meaningful behavior. Humans learn bimanual manipulation skills by watching other humans and by refining their abilities through play. In this work, we aim to enable robots to learn bimanual manipulation behaviors from human video demonstrations and fine-tune them through interaction. Inspired by seminal work in psychology and biomechanics, we propose modeling the interaction between two hands as a serial kinematic linkage -- as a screw motion, in particular, that we use to define a new action space for bimanual manipulation: screw actions. We introduce ScrewMimic, a framework that leverages this novel action representation to facilitate learning from human demonstration and self-supervised policy fine-tuning. Our experiments demonstrate that ScrewMimic is able to learn several complex bimanual behaviors from a single human video demonstration, and that it outperforms baselines that interpret demonstrations and fine-tune directly in the original space of motion of both arms. For more information and video results, https://robin-lab.cs.utexas.edu/ScrewMimic/