# FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

arXiv: https://arxiv.org/abs/2111.05623 · HTML: https://mlsanity.com/paper/2111.05623/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation](https://mlsanity.com/paper/1703.02018/) (2017)
- [FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding](https://mlsanity.com/paper/2105.03655/) (2021)
- [Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks](https://mlsanity.com/paper/2012.03385/) (2020)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [Learning Predictive Representations for Deformable Objects Using Contrastive Estimation](https://mlsanity.com/paper/2003.05436/) (2020)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)
- [Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry](https://mlsanity.com/paper/2104.01167/) (2021)
- [Efficient Bimanual Manipulation Using Learned Task Schemas](https://mlsanity.com/paper/1909.13874/) (2019)
- [Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor](https://mlsanity.com/paper/1910.04854/) (2019)
- [GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion](https://mlsanity.com/paper/2104.05177/) (2021)
- [VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation](https://mlsanity.com/paper/2003.09044/) (2020)
- [Dynamic Cloth Manipulation with Deep Reinforcement Learning](https://mlsanity.com/paper/1910.14475/) (2019)
- [Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation](https://mlsanity.com/paper/2003.08974/) (2020)
- [Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience](https://mlsanity.com/paper/2010.03209/) (2020)
- [FlowControl: Optical Flow Based Visual Servoing](https://mlsanity.com/paper/2007.00291/) (2020)
- [A Grasping-centered Analysis for Cloth Manipulation](https://mlsanity.com/paper/1906.08202/) (2019)
- [Motion Perception in Reinforcement Learning with Dynamic Objects](https://mlsanity.com/paper/1901.03162/) (2019)
- [Intrinsic Motivation for Encouraging Synergistic Behavior](https://mlsanity.com/paper/2002.05189/) (2020)

## Built on by (most cited first, 30 of 54)

- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects](https://mlsanity.com/paper/2205.04382/) (2022)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation](https://mlsanity.com/paper/2601.03782/) (2026)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [SpeedFolding: Learning Efficient Bimanual Folding of Garments](https://mlsanity.com/paper/2208.10552/) (2022)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulation](https://mlsanity.com/paper/2310.16050/) (2023)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation](https://mlsanity.com/paper/2303.11057/) (2023)
- [Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow](https://mlsanity.com/paper/2512.24766/) (2025)
- [DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics](https://mlsanity.com/paper/2304.03223/) (2023)
- [UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence](https://mlsanity.com/paper/2405.06903/) (2024)
- [VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulation](https://mlsanity.com/paper/2407.04152/) (2024)
- [SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulation](https://mlsanity.com/paper/2501.14400/) (2025)
- [Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloning](https://mlsanity.com/paper/2405.18196/) (2024)
- [ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection](https://mlsanity.com/paper/2405.03666/) (2024)
- [DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulation](https://mlsanity.com/paper/2207.01971/) (2022)
- [Neural Eulerian Scene Flow Fields](https://mlsanity.com/paper/2410.02031/) (2024)
- [Normalizing Flows are Capable Models for Continuous Control](https://mlsanity.com/paper/2505.23527/) (2025)
- [UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding](https://mlsanity.com/paper/2311.01267/) (2023)
- [EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamics](https://mlsanity.com/paper/2209.08996/) (2022)
- [Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention](https://mlsanity.com/paper/2301.03003/) (2023)
- [NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation](https://mlsanity.com/paper/2403.08355/) (2024)
- [Autonomously Untangling Long Cables](https://mlsanity.com/paper/2207.07813/) (2022)
- [GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulation](https://mlsanity.com/paper/2503.09243/) (2025)
- [Learning Fabric Manipulation in the Real World with Human Videos](https://mlsanity.com/paper/2211.02832/) (2022)
- [SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy](https://mlsanity.com/paper/2403.10401/) (2024)
- [Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research](https://mlsanity.com/paper/2204.10297/) (2022)

## Abstract (arXiv)

We address the problem of goal-directed cloth manipulation, a challenging task due to the deformability of cloth. Our insight is that optical flow, a technique normally used for motion estimation in video, can also provide an effective representation for corresponding cloth poses across observation and goal images. We introduce FabricFlowNet (FFN), a cloth manipulation policy that leverages flow as both an input and as an action representation to improve performance. FabricFlowNet also elegantly switches between bimanual and single-arm actions based on the desired goal. We show that FabricFlowNet significantly outperforms state-of-the-art model-free and model-based cloth manipulation policies that take image input. We also present real-world experiments on a bimanual system, demonstrating effective sim-to-real transfer. Finally, we show that our method generalizes when trained on a single square cloth to other cloth shapes, such as T-shirts and rectangular cloths. Video and other supplementary materials are available at: https://sites.google.com/view/fabricflownet.