# ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulation

arXiv: https://arxiv.org/abs/2605.27852 · HTML: https://mlsanity.com/paper/2605.27852/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style](https://mlsanity.com/paper/2003.04583/) (2020)
- [Predicting Physics in Mesh-reduced Space with Temporal Attention](https://mlsanity.com/paper/2201.09113/) (2022)
- [SNUG: Self-Supervised Neural Dynamic Garments](https://mlsanity.com/paper/2204.02219/) (2022)
- [Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On](https://mlsanity.com/paper/2105.06462/) (2021)
- [HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics](https://mlsanity.com/paper/2212.07242/) (2022)
- [CROM: Continuous Reduced-Order Modeling of PDEs Using Implicit Neural Representations](https://mlsanity.com/paper/2206.02607/) (2022)
- [Fully Convolutional Graph Neural Networks for Parametric Virtual Try-On](https://mlsanity.com/paper/2009.04592/) (2020)
- [Efficient Learning of Mesh-Based Physical Simulation with BSMS-GNN](https://mlsanity.com/paper/2210.02573/) (2022)
- [DeePSD: Automatic Deep Skinning And Pose Space Deformation For 3D Garment Animation](https://mlsanity.com/paper/2009.02715/) (2020)
- [Neural Cloth Simulation](https://mlsanity.com/paper/2212.11220/) (2022)
- [Generating Datasets of 3D Garments with Sewing Patterns](https://mlsanity.com/paper/2109.05633/) (2021)
- [PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations](https://mlsanity.com/paper/2505.24717/) (2025)
- [ContourCraft: Learning to Resolve Intersections in Neural Multi-Garment Simulations](https://mlsanity.com/paper/2405.09522/) (2024)
- [Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready 3D Characters](https://mlsanity.com/paper/2411.18197/) (2024)
- [RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination](https://mlsanity.com/paper/2505.21925/) (2025)
- [Towards Multi-Layered 3D Garments Animation](https://mlsanity.com/paper/2305.10418/) (2023)
- [Transformer with Implicit Edges for Particle-based Physics Simulation](https://mlsanity.com/paper/2207.10860/) (2022)
- [LiCROM: Linear-Subspace Continuous Reduced Order Modeling with Neural Fields](https://mlsanity.com/paper/2310.15907/) (2023)
- [Neural Garment Dynamics via Manifold-Aware Transformers](https://mlsanity.com/paper/2407.06101/) (2024)
- [A Repulsive Force Unit for Garment Collision Handling in Neural Networks](https://mlsanity.com/paper/2207.13871/) (2022)
- [GIPC: Fast and stable Gauss-Newton optimization of IPC barrier energy](https://mlsanity.com/paper/2308.09400/) (2023)
- [SENC: Handling Self-collision in Neural Cloth Simulation](https://mlsanity.com/paper/2407.12479/) (2024)
- [SwinGar: Spectrum-Inspired Neural Dynamic Deformation for Free-Swinging Garments](https://mlsanity.com/paper/2308.02827/) (2023)

## Built on by (most cited first, 3 of 3)

- [MeshPriorDiT: Hierarchical Modeling for Action-Conditioned Cloth Dynamics](https://mlsanity.com/paper/2608.26766/) (2026)
- [PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation](https://mlsanity.com/paper/2610.07609/) (2026)
- [ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction](https://mlsanity.com/paper/2610.14200/) (2026)

## Abstract (arXiv)

Unified and scalable Transformers have recently achieved remarkable success in modeling diverse phenomena traditionally associated with computer graphics, such as 3D visual effects, rendering processes, and motion in videos. In this work, we take a step further by investigating whether modern Transformer techniques can tackle the challenging task of cloth simulation. To this end, we present ClothTransformer, a framework that reformulates cloth simulation as autoregressive sequence modeling in a learned latent space. Existing neural cloth simulators are largely specialized to single scenarios, intrinsically coupled to the mesh discretization, and lack robust collision handling. Our approach addresses these limitations through three contributions: (1) a unified Transformer architecture that handles diverse scenarios -- body-driven garments, robotic manipulation, and free-fall collisions -- under a single model and achieves approximately 4--9× lower error than prior state-of-the-art methods across all scenarios; (2) a scalable latent-space formulation that compresses arbitrary-resolution meshes into a fixed-size set of latent tokens, making temporal dynamics computation independent of mesh resolution; and (3) a diverse-scenario high-fidelity penetration-free dataset of ~493.4k frames spanning all three settings, which enables a differentiable Continuous Collision Detection (CCD) module to suppress penetration artifacts. Project Page: https://yucrazing.github.io/clothtransformer/