mlsanityData to 7 Oct 2026

VisuoSpatial Foresight for Physical Sequential Fabric Manipulation

arXiv 2102.09754 ↗First posted 19 Feb 2021cs.RO cs.AI cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Robotic fabric manipulation has applications in home robotics, textiles, senior care and surgery. Existing fabric manipulation techniques, however, are designed for specific tasks, making it difficult to generalize across different but related tasks. We build upon the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different sequential fabric manipulation tasks with a single goal-conditioned policy. We extend our earlier work on VisuoSpatial Foresight (VSF), which learns visual dynamics on domain randomized RGB images and depth maps simultaneously and completely in simulation. In this earlier work, we evaluated VSF on multi-step fabric smoothing and folding tasks against 5 baseline methods in simulation and on the da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train or test time. A key finding was that depth sensing significantly improves performance: RGBD data yields an 80% improvement in fabric folding success rate in simulation over pure RGB data. In this work, we vary 4 components of VSF, including data generation, visual dynamics model, cost function, and optimization procedure. Results suggest that training visual dynamics models using longer, corner-based actions can improve the efficiency of fabric folding by 76% and enable a physical sequential fabric folding task that VSF could not previously perform with 90% reliability. Code, data, videos, and supplementary material are available at https://sites.google.com/view/fabric-vsf/.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 48 of its 95 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

48 of 95 references.

References · 48

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2019Learning by Cheatingcited by 361
  8. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  9. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  10. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  11. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  12. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
36 more
  1. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  2. 2018Stochastic Video Generation with a Learned Priorcited by 244
  3. 2017Stochastic Variational Video Predictioncited by 234
  4. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  5. 2018Stochastic Adversarial Video Predictioncited by 173
  6. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  7. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  8. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  9. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 118
  10. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  11. 2015Safe Controller Optimization for Quadrotors with Gaussian Processescited by 77
  12. 2020Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networkscited by 77
  13. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
  14. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  15. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  16. 2020Goal-Aware Prediction: Learning to Model What Matterscited by 45
  17. 2019Assistive Gym: A Physics Simulation Framework for Assistive Roboticscited by 44
  18. 2019Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisorcited by 44
  19. 2020VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulationcited by 40
  20. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  21. 2019Dynamic Cloth Manipulation with Deep Reinforcement Learningcited by 26
  22. 2020Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulationcited by 25
  23. 2020Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experiencecited by 24
  24. 2015Folding Deformable Objects using Predictive Simulation and Trajectory Optimizationcited by 19
  25. 2019Learning How to Autonomously Race a Car: a Predictive Control Approachcited by 14
  26. 2019A Grasping-centered Analysis for Cloth Manipulationcited by 14
  27. 2017Cautious NMPC with Gaussian Process Dynamics for Autonomous Miniature Race Carscited by 13
  28. 2017Deep Haptic Model Predictive Control for Robot-Assisted Dressingcited by 12
  29. 2019Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Controlcited by 12
  30. 2017Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedurecited by 11
  31. 2019On-Policy Robot Imitation Learning from a Converging Supervisorcited by 10
  32. 2018Cloth Manipulation Using Random-Forest-Based Imitation Learningcited by 7
  33. 2017Manipulating Highly Deformable Materials Using a Visual Feedback Dictionarycited by 5
  34. 2016Multi-Sensor Surface Analysis for Robotic Ironingcited by 4
  35. 2018Time Reversal as Self-Supervisioncited by 4
  36. 2020MMGSD: Multi-Modal Gaussian Shape Descriptors for Correspondence Matching in 1D and 2D Deformable Objectscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 4; 2024: 3; 2025: 4.

18 papers.

Sort:

By area:

  1. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
  2. 2022ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulationcited by 24
  3. 2023DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physicscited by 23
  4. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  5. 2025Diffusion Dynamics Models with Generative State Estimation for Cloth Manipulationcited by 16
  6. 2022Contact Points Discovery for Soft-Body Manipulations with Differentiable Physicscited by 14
  7. 2022EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamicscited by 10
  8. 2025Cloth-Splatting: 3D Cloth State Estimation from RGB Supervisioncited by 7
  9. 2023DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physicscited by 6
  10. 2022Safe Deep RL in 3D Environments using Human Feedbackcited by 5
  11. 2022Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactionscited by 5
  12. 2023PlaNet-ClothPick: Effective Fabric Flattening Based on Latent Dynamic Planningcited by 1
6 more
  1. 2023Learning to bag with a simulation-free reinforcement learning framework for robotscited by 1
  2. 2024Implicit Contact Diffuser: Sequential Contact Reasoning with Latent Point Cloud Diffusioncited by 1
  3. 2025Learning a General Model: Folding Clothing with Topological Dynamicscited by 1
  4. 2025CLASP: General-Purpose Clothes Manipulation with Semantic Keypointscited by 1
  5. 2024Autonomous Quilt Spreading for Caregiving Robotscited by 0
  6. 2024DRAPER: Towards a Robust Robot Deployment and Reliable Evaluation for Quasi-Static Pick-and-Place Cloth-Shaping Neural Controllerscited by 0