# ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction

arXiv: https://arxiv.org/abs/2610.14200 · HTML: https://mlsanity.com/paper/2610.14200/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023)
- [PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos](https://mlsanity.com/paper/2503.17973/) (2025)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [Learning Visible Connectivity Dynamics for Cloth Smoothing](https://mlsanity.com/paper/2105.10389/) (2021)
- [AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation](https://mlsanity.com/paper/2407.07889/) (2024)
- [GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion](https://mlsanity.com/paper/2104.05177/) (2021)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos](https://mlsanity.com/paper/2506.15680/) (2025)
- [Diffusion Dynamics Models with Generative State Estimation for Cloth Manipulation](https://mlsanity.com/paper/2503.11999/) (2025)
- [Unfolding the Literature: A Review of Robotic Cloth Manipulation](https://mlsanity.com/paper/2407.01361/) (2024)
- [Semantic State Estimation in Cloth Manipulation Tasks](https://mlsanity.com/paper/2203.11647/) (2022)
- [Masked Generative Policy for Robotic Control](https://mlsanity.com/paper/2512.09101/) (2025)
- [ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulation](https://mlsanity.com/paper/2605.27852/) (2026)

## Abstract (arXiv)

We introduce ClothCode, a unified generative framework for garment state estimation and action-conditioned dynamics prediction in a shared discrete configuration space. A material-aligned tokenizer encodes garment configuration as residual vector-quantized patch tokens, while task-specific Masked Generative Transformers predict these tokens through parallel confidence-guided refinement. Across three garment templates, ClothCode reduces mean displacement error by 26.6% for state estimation and 37.0% for dynamics prediction relative to the strongest baselines. Dynamics prediction requires only 16.7 ms per window, compared with 556.2 ms for diffusion. Integrated into the same model-predictive controller, ClothCode improves real-world folding and flattening success from 56.7% to 78.9%. Further results and videos are available at: https://anonymous.4open.science/r/ClothCode-DE32