mlsanityData to 7 Oct 2026

Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking

arXiv 2302.09502 ↗First posted 19 Feb 2023cs.RO cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

State estimation is one of the greatest challenges for cloth manipulation due to cloth's high dimensionality and self-occlusion. Prior works propose to identify the full state of crumpled clothes by training a mesh reconstruction model in simulation. However, such models are prone to suffer from a sim-to-real gap due to differences between cloth simulation and the real world. In this work, we propose a self-supervised method to finetune a mesh reconstruction model in the real world. Since the full mesh of crumpled cloth is difficult to obtain in the real world, we design a special data collection scheme and an action-conditioned model-based cloth tracking method to generate pseudo-labels for self-supervised learning. By finetuning the pretrained mesh reconstruction model on this pseudo-labeled dataset, we show that we can improve the quality of the reconstructed mesh without requiring human annotations, and improve the performance of downstream manipulation task.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 31 of its 66 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

31 of 66 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  3. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  6. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  7. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  8. 2019Solving Rubik's Cube with a Robot Handcited by 699
  9. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  10. 2019Multi-Garment Net: Learning to Dress 3D People from Imagescited by 244
  11. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  12. 2021SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networkscited by 181
19 more
  1. 2020TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Stylecited by 148
  2. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  3. 2020BCNet: Learning Body and Cloth Shape from A Single Imagecited by 118
  4. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 118
  5. 2019CLOTH3D: Clothed 3D Humanscited by 114
  6. 2020Deep Fashion3D: A Dataset and Benchmark for 3D Garment Reconstruction from Single Imagescited by 101
  7. 2015Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Viewscited by 94
  8. 2021FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfoldingcited by 80
  9. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
  10. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  11. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  12. 2020DeepCloth: Neural Garment Representation for Shape and Style Editingcited by 54
  13. 2019Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisorcited by 44
  14. 2021GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completioncited by 41
  15. 2022Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulationcited by 32
  16. 2022DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objectscited by 29
  17. 2015Folding Deformable Objects using Predictive Simulation and Trajectory Optimizationcited by 19
  18. 2021Occlusion-robust Deformable Object Tracking without Physics Simulationcited by 15
  19. 2020Tracking Partially-Occluded Deformable Objects while Enforcing Geometric Constraintscited by 12

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 3; 2025: 1.

8 papers.

  1. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 59
  2. 2023Benchmarking the Sim-to-Real Gap in Cloth Manipulationcited by 14
  3. 2023SculptBot: Pre-Trained Models for 3D Deformable Object Manipulationcited by 8
  4. 2025Cloth-Splatting: 3D Cloth State Estimation from RGB Supervisioncited by 7
  5. 2024TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approachcited by 5
  6. 2024ClothPPO: A Proximal Policy Optimization Enhancing Framework for Robotic Cloth Manipulation with Observation-Aligned Action Spacescited by 2
  7. 2023PlaNet-ClothPick: Effective Fabric Flattening Based on Latent Dynamic Planningcited by 1
  8. 2024RaggeDi: Diffusion-based State Estimation of Disordered Rags, Sheets, Towels and Blanketscited by 1