# Reactive In-Air Clothing Manipulation with Confidence-Aware Dense Correspondence and Visuotactile Affordance

arXiv: https://arxiv.org/abs/2509.03889 · HTML: https://mlsanity.com/paper/2509.03889/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [ALOHA Unleashed: A Simple Recipe for Robot Dexterity](https://mlsanity.com/paper/2410.13126/) (2024)
- [FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding](https://mlsanity.com/paper/2105.03655/) (2021)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [Learning Visible Connectivity Dynamics for Cloth Smoothing](https://mlsanity.com/paper/2105.10389/) (2021)
- [Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data](https://mlsanity.com/paper/2003.01835/) (2020)
- [GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion](https://mlsanity.com/paper/2104.05177/) (2021)
- [VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation](https://mlsanity.com/paper/2003.09044/) (2020)
- [GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger](https://mlsanity.com/paper/2106.08851/) (2021)
- [UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence](https://mlsanity.com/paper/2405.06903/) (2024)
- [Visuotactile Affordances for Cloth Manipulation with Local Control](https://mlsanity.com/paper/2212.05108/) (2022)
- [Cloth Region Segmentation for Robust Grasp Selection](https://mlsanity.com/paper/2008.05626/) (2020)
- [Occlusion-robust Deformable Object Tracking without Physics Simulation](https://mlsanity.com/paper/2101.00733/) (2021)
- [Learning to Grasp Clothing Structural Regions for Garment Manipulation Tasks](https://mlsanity.com/paper/2306.14553/) (2023)
- [Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics](https://mlsanity.com/paper/2003.12698/) (2020)
- [Force-Constrained Visual Policy: Safe Robot-Assisted Dressing via Multi-Modal Sensing](https://mlsanity.com/paper/2311.04390/) (2023)
- [Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research](https://mlsanity.com/paper/2204.10297/) (2022)
- [RoboHanger: Learning Generalizable Robotic Hanger Insertion for Diverse Garments](https://mlsanity.com/paper/2412.01083/) (2024)
- [GraphGarment: Learning Garment Dynamics for Bimanual Cloth Manipulation Tasks](https://mlsanity.com/paper/2503.05817/) (2025)
- [MMGSD: Multi-Modal Gaussian Shape Descriptors for Correspondence Matching in 1D and 2D Deformable Objects](https://mlsanity.com/paper/2010.04339/) (2020)

## Built on by (most cited first, 4 of 4)

- [OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer](https://mlsanity.com/paper/2512.08920/) (2025)
- [Instant-Fold: In-Context Imitation Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2606.04269/) (2026)
- [TRACER: Texture-Robust Affordance Chain-of-Thought for Deformable-Object Refinement](https://mlsanity.com/paper/2601.20208/) (2026)
- [Vision-Based 6-DoF Grasp Pose Estimation for Robot Cloth Unfolding](https://mlsanity.com/paper/2609.31452/) (2026)

## Abstract (arXiv)

Manipulating clothing is challenging due to complex configurations, variable material dynamics, and frequent self-occlusion. Prior systems often flatten garments or assume visibility of key features. We present a dual-arm visuotactile framework that combines confidence-aware dense visual correspondence and tactile-supervised grasp affordance to operate directly on crumpled and suspended garments. The correspondence model is trained on a custom, high-fidelity simulated dataset using a distributional loss that captures cloth symmetries and generates correspondence confidence estimates. These estimates guide a reactive state machine that adapts folding strategies based on perceptual uncertainty. In parallel, a visuotactile grasp affordance network, self-supervised using high-resolution tactile feedback, determines which regions are physically graspable. The same tactile classifier is used during execution for real-time grasp validation. By deferring action in low-confidence states, the system handles highly occluded table-top and in-air configurations. We demonstrate our task-agnostic grasp selection module in folding and hanging tasks. Moreover, our dense descriptors provide a reusable intermediate representation for other planning modalities, such as extracting grasp targets from human video demonstrations, paving the way for more generalizable and scalable garment manipulation.