# Flat'n'Fold: A Diverse Multi-Modal Dataset for Garment Perception and Manipulation

arXiv: https://arxiv.org/abs/2409.18297 · HTML: https://mlsanity.com/paper/2409.18297/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021)
- [DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data](https://mlsanity.com/paper/1912.04302/) (2019)
- [Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation](https://mlsanity.com/paper/1810.07121/) (2018)
- [SpeedFolding: Learning Efficient Bimanual Folding of Garments](https://mlsanity.com/paper/2208.10552/) (2022)
- [Universal Visual Decomposer: Long-Horizon Manipulation Made Easy](https://mlsanity.com/paper/2310.08581/) (2023)
- [Quo Vadis, Skeleton Action Recognition ?](https://mlsanity.com/paper/2007.02072/) (2020)
- [GarmentTracking: Category-Level Garment Pose Tracking](https://mlsanity.com/paper/2303.13913/) (2023)
- [A Virtual Reality Framework for Human-Robot Collaboration in Cloth Folding](https://mlsanity.com/paper/2305.07493/) (2023)
- [Semantic State Estimation in Cloth Manipulation Tasks](https://mlsanity.com/paper/2203.11647/) (2022)
- [GarNet: A Continuous Robot Vision Approach for Predicting Shapes and Visually Perceived Weights of Garments](https://mlsanity.com/paper/2109.07831/) (2021)
- [IMMERTWIN: A Mixed Reality Framework for Enhanced Robotic Arm Teleoperation](https://mlsanity.com/paper/2409.08964/) (2024)
- [A Data-Centric Approach For Dual-Arm Robotic Garment Flattening](https://mlsanity.com/paper/2208.13695/) (2022)
- [TELESIM: A Modular and Plug-and-Play Framework for Robotic Arm Teleoperation using a Digital Twin](https://mlsanity.com/paper/2309.10579/) (2023)

## Built on by (most cited first, 1 of 1)

- [TRACER: Texture-Robust Affordance Chain-of-Thought for Deformable-Object Refinement](https://mlsanity.com/paper/2601.20208/) (2026)

## Abstract (arXiv)

We present Flat'n'Fold, a novel large-scale dataset for garment manipulation that addresses critical gaps in existing datasets. Comprising 1,212 human and 887 robot demonstrations of flattening and folding 44 unique garments across 8 categories, Flat'n'Fold surpasses prior datasets in size, scope, and diversity. Our dataset uniquely captures the entire manipulation process from crumpled to folded states, providing synchronized multi-view RGB-D images, point clouds, and action data, including hand or gripper positions and rotations. We quantify the dataset's diversity and complexity compared to existing benchmarks and show that our dataset features natural and diverse manipulations of real-world demonstrations of human and robot demonstrations in terms of visual and action information. To showcase Flat'n'Fold's utility, we establish new benchmarks for grasping point prediction and subtask decomposition. Our evaluation of state-of-the-art models on these tasks reveals significant room for improvement. This underscores Flat'n'Fold's potential to drive advances in robotic perception and manipulation of deformable objects. Our dataset can be downloaded at https://cvas-ug.github.io/flat-n-fold