# GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation

arXiv: https://arxiv.org/abs/2411.01200 · HTML: https://mlsanity.com/paper/2411.01200/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 50 of its 72 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 50)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills](https://mlsanity.com/paper/2302.04659/) (2023)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation](https://mlsanity.com/paper/2305.12821/) (2023)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [CLOTH3D: Clothed 3D Humans](https://mlsanity.com/paper/1912.02792/) (2019)
- [3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations](https://mlsanity.com/paper/2403.03954/) (2024)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols](https://mlsanity.com/paper/1502.03143/) (2015)
- [AKB-48: A Real-World Articulated Object Knowledge Base](https://mlsanity.com/paper/2202.08432/) (2022)
- [FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding](https://mlsanity.com/paper/2105.03655/) (2021)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)

## Built on by (most cited first, 22 of 22)

- [ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning](https://mlsanity.com/paper/2503.21860/) (2025)
- [RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning](https://mlsanity.com/paper/2504.18904/) (2025)
- [RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies](https://mlsanity.com/paper/2607.04434/) (2026)
- [ET-SEED: Efficient Trajectory-Level SE(3) Equivariant Diffusion Policy](https://mlsanity.com/paper/2411.03990/) (2024)
- [GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulation](https://mlsanity.com/paper/2503.09243/) (2025)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [Progress Reward Modeling for Robotic Learning: A Comprehensive Survey](https://mlsanity.com/paper/2607.21655/) (2026)
- [Hierarchical DLO Routing with Reinforcement Learning and In-Context Vision-language Models](https://mlsanity.com/paper/2510.19268/) (2025)
- [LeHome: A Simulation Environment for Deformable Object Manipulation in Household Scenarios](https://mlsanity.com/paper/2604.22363/) (2026)
- [SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects (Early Version)](https://mlsanity.com/paper/2607.04234/) (2026)
- [FoldNet: Learning Generalizable Closed-Loop Policy for Garment Folding via Keypoint-Driven Asset and Demonstration Synthesis](https://mlsanity.com/paper/2505.09109/) (2025)
- [GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning](https://mlsanity.com/paper/2603.04158/) (2026)
- [SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds](https://mlsanity.com/paper/2604.08544/) (2026)
- [From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation](https://mlsanity.com/paper/2604.15805/) (2026)
- [3D Generation for Embodied AI and Robotic Simulation: A Survey](https://mlsanity.com/paper/2604.26509/) (2026)
- [FCBV-Net: Category-Level Robotic Garment Smoothing via Feature-Conditioned Bimanual Value Prediction](https://mlsanity.com/paper/2508.05153/) (2025)
- [TRACER: Texture-Robust Affordance Chain-of-Thought for Deformable-Object Refinement](https://mlsanity.com/paper/2601.20208/) (2026)
- [One Pool Is Not Enough: Multi-Cluster Memory for Practical Test-Time Adaptation](https://mlsanity.com/paper/2603.21135/) (2026)
- [HeteroGenManip: Generalizable Manipulation For Heterogeneous Object Interactions](https://mlsanity.com/paper/2605.10201/) (2026)
- [AnnotateAnything: Automatic Annotation of 3D Assets for Robot Manipulation](https://mlsanity.com/paper/2606.17446/) (2026)
- [APT: Atomic Physical Transitions for Causal Video-Language Understanding](https://mlsanity.com/paper/2606.18586/) (2026)
- [PhySGA: A Physics-Space Grounded Agentic Framework for Generalizable and Interactive 3D Composition](https://mlsanity.com/paper/2610.20560/) (2026)

## Abstract (arXiv)

Manipulating garments and fabrics has long been a critical endeavor in the development of home-assistant robots. However, due to complex dynamics and topological structures, garment manipulations pose significant challenges. Recent successes in reinforcement learning and vision-based methods offer promising avenues for learning garment manipulation. Nevertheless, these approaches are severely constrained by current benchmarks, which offer limited diversity of tasks and unrealistic simulation behavior. Therefore, we present GarmentLab, a content-rich benchmark and realistic simulation designed for deformable object and garment manipulation. Our benchmark encompasses a diverse range of garment types, robotic systems and manipulators. The abundant tasks in the benchmark further explores of the interactions between garments, deformable objects, rigid bodies, fluids, and human body. Moreover, by incorporating multiple simulation methods such as FEM and PBD, along with our proposed sim-to-real algorithms and real-world benchmark, we aim to significantly narrow the sim-to-real gap. We evaluate state-of-the-art vision methods, reinforcement learning, and imitation learning approaches on these tasks, highlighting the challenges faced by current algorithms, notably their limited generalization capabilities. Our proposed open-source environments and comprehensive analysis show promising boost to future research in garment manipulation by unlocking the full potential of these methods. We guarantee that we will open-source our code as soon as possible. You can watch the videos in supplementary files to learn more about the details of our work. Our project page is available at: https://garmentlab.github.io/