# Benchmarking the Sim-to-Real Gap in Cloth Manipulation

arXiv: https://arxiv.org/abs/2310.09543 · HTML: https://mlsanity.com/paper/2310.09543/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 14 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding](https://mlsanity.com/paper/2105.03655/) (2021)
- [Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks](https://mlsanity.com/paper/2012.03385/) (2020)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [Learning Predictive Representations for Deformable Objects Using Contrastive Estimation](https://mlsanity.com/paper/2003.05436/) (2020)
- [Planar Robot Casting with Real2Sim2Real Self-Supervised Learning](https://mlsanity.com/paper/2111.04814/) (2021)
- [Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects](https://mlsanity.com/paper/2203.00663/) (2022)
- [DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects](https://mlsanity.com/paper/2204.03139/) (2022)
- [Dynamic Cloth Manipulation with Deep Reinforcement Learning](https://mlsanity.com/paper/1910.14475/) (2019)
- [DaXBench: Benchmarking Deformable Object Manipulation with Differentiable Physics](https://mlsanity.com/paper/2210.13066/) (2022)
- [Learning Deformable Object Manipulation from Expert Demonstrations](https://mlsanity.com/paper/2207.10148/) (2022)
- [Learning to Collaborate from Simulation for Robot-Assisted Dressing](https://mlsanity.com/paper/1909.06682/) (2019)
- [A Grasping-centered Analysis for Cloth Manipulation](https://mlsanity.com/paper/1906.08202/) (2019)
- [Validating Robotics Simulators on Real-World Impacts](https://mlsanity.com/paper/2110.00541/) (2021)
- [Learning Visual Feedback Control for Dynamic Cloth Folding](https://mlsanity.com/paper/2109.04771/) (2021)
- [Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking](https://mlsanity.com/paper/2302.09502/) (2023)
- [Household Cloth Object Set: Fostering Benchmarking in Deformable Object Manipulation](https://mlsanity.com/paper/2111.01527/) (2021)
- [Benchmarking Simulated Robotic Manipulation through a Real World Dataset](https://mlsanity.com/paper/1911.01557/) (2019)
- [QDP: Learning to Sequentially Optimise Quasi-Static and Dynamic Manipulation Primitives for Robotic Cloth Manipulation](https://mlsanity.com/paper/2303.13320/) (2023)

## Built on by (most cited first, 17 of 17)

- [LLM-as-a-Verifier: A General-Purpose Verification Framework](https://mlsanity.com/paper/2607.05391/) (2026)
- [A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI](https://mlsanity.com/paper/2505.01458/) (2025)
- [Learning Keypoints for Robotic Cloth Manipulation using Synthetic Data](https://mlsanity.com/paper/2401.01734/) (2024)
- [GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models](https://mlsanity.com/paper/2608.05948/) (2026)
- [D-Cubed: Latent Diffusion Trajectory Optimisation for Dexterous Deformable Manipulation](https://mlsanity.com/paper/2403.12861/) (2024)
- [BiFold: Bimanual Cloth Folding with Language Guidance](https://mlsanity.com/paper/2501.16458/) (2025)
- [Benchmarking Model Predictive Control and Reinforcement Learning Based Control for Legged Robot Locomotion in MuJoCo Simulation](https://mlsanity.com/paper/2501.16590/) (2025)
- [Robustness without Wrinkles: Parallel Simulation and Robust MPC for Certified Deformable Manipulation](https://mlsanity.com/paper/2606.14188/) (2026)
- [Implicit Neural-Representation Learning for Elastic Deformable-Object Manipulations](https://mlsanity.com/paper/2505.00500/) (2025)
- [Position: Intelligent Science Laboratory Requires the Integration of Cognitive and Embodied AI](https://mlsanity.com/paper/2506.19613/) (2025)
- [Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warping](https://mlsanity.com/paper/2603.03278/) (2026)
- [Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-Shot Real-World Deployment](https://mlsanity.com/paper/2603.28475/) (2026)
- [Dynamic Manipulation of Deformable Objects using Imitation Learning with Adaptation to Hardware Constraints](https://mlsanity.com/paper/2403.12685/) (2024)
- [An Open-source Sim2Real Approach for Sensor-independent Robot Navigation in a Grid](https://mlsanity.com/paper/2411.03494/) (2024)
- [Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control](https://mlsanity.com/paper/2605.18373/) (2026)
- [VFILC: Accurate Frequency Extrapolations in Imitation Learning via Sampling Frequency ILC](https://mlsanity.com/paper/2606.20056/) (2026)
- [PAKT: Physically-Aligned Kinesthetic Teaching for Reinforcement Learning](https://mlsanity.com/paper/2609.25630/) (2026)

## Abstract (arXiv)

Realistic physics engines play a crucial role for learning to manipulate deformable objects such as garments in simulation. By doing so, researchers can circumvent challenges such as sensing the deformation of the object in the realworld. In spite of the extensive use of simulations for this task, few works have evaluated the reality gap between deformable object simulators and real-world data. We present a benchmark dataset to evaluate the sim-to-real gap in cloth manipulation. The dataset is collected by performing a dynamic as well as a quasi-static cloth manipulation task involving contact with a rigid table. We use the dataset to evaluate the reality gap, computational time, and simulation stability of four popular deformable object simulators: MuJoCo, Bullet, Flex, and SOFA. Additionally, we discuss the benefits and drawbacks of each simulator. The benchmark dataset is open-source. Supplementary material, videos, and code, can be found at https://sites.google.com/view/cloth-sim2real-benchmark.