# Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor

arXiv: https://arxiv.org/abs/1910.04854 · HTML: https://mlsanity.com/paper/1910.04854/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 13 of its 57 references that are in the map.
- First posted on arXiv 23 Sep 2019.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Folding Deformable Objects using Predictive Simulation and Trajectory Optimization](https://mlsanity.com/paper/1512.06922/) (2015)
- [A Grasping-centered Analysis for Cloth Manipulation](https://mlsanity.com/paper/1906.08202/) (2019)
- [Deep Haptic Model Predictive Control for Robot-Assisted Dressing](https://mlsanity.com/paper/1709.09735/) (2017)
- [Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedure](https://mlsanity.com/paper/1709.06668/) (2017)
- [Multi-Sensor Surface Analysis for Robotic Ironing](https://mlsanity.com/paper/1602.04918/) (2016)
- [Tracking Human Pose During Robot-Assisted Dressing using Single-Axis Capacitive Proximity Sensing](https://mlsanity.com/paper/1709.07957/) (2017)

## Built on by (most cited first, 30 of 44)

- [Learning Visible Connectivity Dynamics for Cloth Smoothing](https://mlsanity.com/paper/2105.10389/) (2021)
- [FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy](https://mlsanity.com/paper/2111.05623/) (2021)
- [SpeedFolding: Learning Efficient Bimanual Folding of Garments](https://mlsanity.com/paper/2208.10552/) (2022)
- [LazyDAgger: Reducing Context Switching in Interactive Imitation Learning](https://mlsanity.com/paper/2104.00053/) (2021)
- [Physical Design using Differentiable Learned Simulators](https://mlsanity.com/paper/2202.00728/) (2022)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence](https://mlsanity.com/paper/2405.06903/) (2024)
- [VisuoSpatial Foresight for Physical Sequential Fabric Manipulation](https://mlsanity.com/paper/2102.09754/) (2021)
- [SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo](https://mlsanity.com/paper/2106.16118/) (2021)
- [Imitation Learning for Nonprehensile Manipulation through Self-Supervised Learning Considering Motion Speed](https://mlsanity.com/paper/2206.10289/) (2022)
- [A Survey on Imitation Learning for Contact-Rich Tasks in Robotics](https://mlsanity.com/paper/2506.13498/) (2025)
- [Learning Representations that Enable Generalization in Assistive Tasks](https://mlsanity.com/paper/2212.03175/) (2022)
- [Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies](https://mlsanity.com/paper/2107.08942/) (2021)
- [A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation](https://mlsanity.com/paper/2112.05068/) (2021)
- [Learning Keypoints for Robotic Cloth Manipulation using Synthetic Data](https://mlsanity.com/paper/2401.01734/) (2024)
- [Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention](https://mlsanity.com/paper/2301.03003/) (2023)
- [Robotic Compliant Object Prying Using Diffusion Policy Guided by Vision and Force Observations](https://mlsanity.com/paper/2503.03998/) (2025)
- [Autonomously Untangling Long Cables](https://mlsanity.com/paper/2207.07813/) (2022)
- [Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking](https://mlsanity.com/paper/2302.09502/) (2023)
- [Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision](https://mlsanity.com/paper/2501.01715/) (2025)
- [Learning Fabric Manipulation in the Real World with Human Videos](https://mlsanity.com/paper/2211.02832/) (2022)
- [Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research](https://mlsanity.com/paper/2204.10297/) (2022)
- [Learning Sequential Acquisition Policies for Robot-Assisted Feeding](https://mlsanity.com/paper/2309.05197/) (2023)
- [Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactions](https://mlsanity.com/paper/2208.05632/) (2022)
- [SGTM 2.0: Autonomously Untangling Long Cables using Interactive Perception](https://mlsanity.com/paper/2209.13706/) (2022)
- [GOATS: Goal Sampling Adaptation for Scooping with Curriculum Reinforcement Learning](https://mlsanity.com/paper/2303.05193/) (2023)
- [Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2511.11298/) (2025)
- [Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap](https://mlsanity.com/paper/2103.02554/) (2021)
- [IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors](https://mlsanity.com/paper/2306.15228/) (2023)
- [Multi-objective Cross-task Learning via Goal-conditioned GPT-based Decision Transformers for Surgical Robot Task Automation](https://mlsanity.com/paper/2405.18757/) (2024)

## Abstract (arXiv)

Sequential pulling policies to flatten and smooth fabrics have applications from surgery to manufacturing to home tasks such as bed making and folding clothes. Due to the complexity of fabric states and dynamics, we apply deep imitation learning to learn policies that, given color (RGB), depth (D), or combined color-depth (RGBD) images of a rectangular fabric sample, estimate pick points and pull vectors to spread the fabric to maximize coverage. To generate data, we develop a fabric simulator and an algorithmic supervisor that has access to complete state information. We train policies in simulation using domain randomization and dataset aggregation (DAgger) on three tiers of difficulty in the initial randomized configuration. We present results comparing five baseline policies to learned policies and report systematic comparisons of RGB vs D vs RGBD images as inputs. In simulation, learned policies achieve comparable or superior performance to analytic baselines. In 180 physical experiments with the da Vinci Research Kit (dVRK) surgical robot, RGBD policies trained in simulation attain coverage of 83% to 95% depending on difficulty tier, suggesting that effective fabric smoothing policies can be learned from an algorithmic supervisor and that depth sensing is a valuable addition to color alone. Supplementary material is available at https://sites.google.com/view/fabric-smoothing.