# Learning Fabric Manipulation in the Real World with Human Videos

arXiv: https://arxiv.org/abs/2211.02832 · HTML: https://mlsanity.com/paper/2211.02832/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 5 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube](https://mlsanity.com/paper/2202.10448/) (2022)
- [FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding](https://mlsanity.com/paper/2105.03655/) (2021)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [Learning Predictive Representations for Deformable Objects Using Contrastive Estimation](https://mlsanity.com/paper/2003.05436/) (2020)
- [Equivariant Q Learning in Spatial Action Spaces](https://mlsanity.com/paper/2110.15443/) (2021)
- [FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy](https://mlsanity.com/paper/2111.05623/) (2021)
- [SpeedFolding: Learning Efficient Bimanual Folding of Garments](https://mlsanity.com/paper/2208.10552/) (2022)
- [Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor](https://mlsanity.com/paper/1910.04854/) (2019)
- [VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation](https://mlsanity.com/paper/2003.09044/) (2020)
- [Dynamic Cloth Manipulation with Deep Reinforcement Learning](https://mlsanity.com/paper/1910.14475/) (2019)
- [Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience](https://mlsanity.com/paper/2010.03209/) (2020)
- [DextAIRity: Deformable Manipulation Can be a Breeze](https://mlsanity.com/paper/2203.01197/) (2022)
- [Learning Visual Feedback Control for Dynamic Cloth Folding](https://mlsanity.com/paper/2109.04771/) (2021)
- [Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research](https://mlsanity.com/paper/2204.10297/) (2022)
- [Multi-Sensor Surface Analysis for Robotic Ironing](https://mlsanity.com/paper/1602.04918/) (2016)
- [Learning Switching Criteria for Sim2Real Transfer of Robotic Fabric Manipulation Policies](https://mlsanity.com/paper/2207.00911/) (2022)
- [Demonstrating Cloth Folding to Robots: Design and Evaluation of a 2D and a 3D User Interface](https://mlsanity.com/paper/2104.02968/) (2021)

## Built on by (most cited first, 7 of 7)

- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation](https://mlsanity.com/paper/2303.11057/) (2023)
- [VIEW: Visual Imitation Learning with Waypoints](https://mlsanity.com/paper/2404.17906/) (2024)
- [Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations](https://mlsanity.com/paper/2307.05959/) (2023)
- [Learn2Fold: Structured Origami Generation with World Model Planning](https://mlsanity.com/paper/2603.29585/) (2026)
- [One-Shot Affordance Grounding of Deformable Objects in Egocentric Organizing Scenes](https://mlsanity.com/paper/2503.01092/) (2025)
- [CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints](https://mlsanity.com/paper/2507.19983/) (2025)

## Abstract (arXiv)

Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Learning approaches stand out as promising for this domain as they allow us to learn behaviours directly from data. Most prior methods however rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects or rely on large datasets. A promising alternative is to learn fabric manipulation directly from watching humans perform the task. In this work, we explore how demonstrations for fabric manipulation tasks can be collected directly by humans, providing an extremely natural and fast data collection pipeline. Then, using only a handful of such demonstrations, we show how a pick-and-place policy can be learned and deployed on a real robot, without any robot data collection at all. We demonstrate our approach on a fabric folding task, showing that our policy can reliably reach folded states from crumpled initial configurations. Videos are available at: https://sites.google.com/view/foldingbyhand