# Semantic State Estimation in Cloth Manipulation Tasks

arXiv: https://arxiv.org/abs/2203.11647 · HTML: https://mlsanity.com/paper/2203.11647/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks](https://mlsanity.com/paper/2012.03385/) (2020)
- [Learning Predictive Representations for Deformable Objects Using Contrastive Estimation](https://mlsanity.com/paper/2003.05436/) (2020)
- [Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor](https://mlsanity.com/paper/1910.04854/) (2019)
- [VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation](https://mlsanity.com/paper/2003.09044/) (2020)
- [Dynamic Cloth Manipulation with Deep Reinforcement Learning](https://mlsanity.com/paper/1910.14475/) (2019)
- [Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation](https://mlsanity.com/paper/2003.08974/) (2020)
- [Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View](https://mlsanity.com/paper/1809.10305/) (2018)
- [A Grasping-centered Analysis for Cloth Manipulation](https://mlsanity.com/paper/1906.08202/) (2019)
- [Model-Driven Feed-Forward Prediction for Manipulation of Deformable Objects](https://mlsanity.com/paper/1607.04411/) (2016)
- [Fashion Landmark Detection and Category Classification for Robotics](https://mlsanity.com/paper/2003.11827/) (2020)
- [Bodies Uncovered: Learning to Manipulate Real Blankets Around People via Physics Simulations](https://mlsanity.com/paper/2109.04930/) (2021)
- [Grasp-Oriented Fine-grained Cloth Segmentation without Real Supervision](https://mlsanity.com/paper/2110.02903/) (2021)
- [Single-Shot Clothing Category Recognition in Free-Configurations with Application to Autonomous Clothes Sorting](https://mlsanity.com/paper/1707.07157/) (2017)
- [Sequential Topological Representations for Predictive Models of Deformable Objects](https://mlsanity.com/paper/2011.11693/) (2020)
- [An Energy Minimization Approach to 3D Non-Rigid Deformable Surface Estimation Using RGBD Data](https://mlsanity.com/paper/1708.00940/) (2017)
- [Deep Shape-from-Template: Wide-Baseline, Dense and Fast Registration and Deformable Reconstruction from a Single Image](https://mlsanity.com/paper/1811.07791/) (2018)
- [Encoding cloth manipulations using a graph of states and transitions](https://mlsanity.com/paper/2009.14681/) (2020)

## Built on by (most cited first, 3 of 3)

- [Flat'n'Fold: A Diverse Multi-Modal Dataset for Garment Perception and Manipulation](https://mlsanity.com/paper/2409.18297/) (2024)
- [Leveraging Foundation Models To learn the shape of semi-fluid deformable objects](https://mlsanity.com/paper/2411.16802/) (2024)
- [ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction](https://mlsanity.com/paper/2610.14200/) (2026)

## Abstract (arXiv)

Understanding of deformable object manipulations such as textiles is a challenge due to the complexity and high dimensionality of the problem. Particularly, the lack of a generic representation of semantic states (e.g., textitcrumpled, textitdiagonally folded) during a continuous manipulation process introduces an obstacle to identify the manipulation type. In this paper, we aim to solve the problem of semantic state estimation in cloth manipulation tasks. For this purpose, we introduce a new large-scale fully-annotated RGB image dataset showing various human demonstrations of different complicated cloth manipulations. We provide a set of baseline deep networks and benchmark them on the problem of semantic state estimation using our proposed dataset. Furthermore, we investigate the scalability of our semantic state estimation framework in robot monitoring tasks of long and complex cloth manipulations.