# ViTac-Tracing: Visual-Tactile Imitation Learning of Deformable Object Tracing

arXiv: https://arxiv.org/abs/2603.18784 · HTML: https://mlsanity.com/paper/2603.18784/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 20 of 20)

- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators](https://mlsanity.com/paper/2309.13037/) (2023)
- [PLAS: Latent Action Space for Offline Reinforcement Learning](https://mlsanity.com/paper/2011.07213/) (2020)
- [Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation](https://mlsanity.com/paper/2503.02881/) (2025)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [Challenges and Outlook in Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2105.01767/) (2021)
- [GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger](https://mlsanity.com/paper/2106.08851/) (2021)
- [Cloth Funnels: Canonicalized-Alignment for Multi-Purpose Garment Manipulation](https://mlsanity.com/paper/2210.09347/) (2022)
- [Visuotactile Affordances for Cloth Manipulation with Local Control](https://mlsanity.com/paper/2212.05108/) (2022)
- [Learning Keypoints for Robotic Cloth Manipulation using Synthetic Data](https://mlsanity.com/paper/2401.01734/) (2024)
- [Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning](https://mlsanity.com/paper/2204.00117/) (2022)
- [Autonomously Untangling Long Cables](https://mlsanity.com/paper/2207.07813/) (2022)
- [ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations](https://mlsanity.com/paper/2506.20757/) (2025)
- [In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning](https://mlsanity.com/paper/2103.09402/) (2021)
- [Precise Robotic Needle-Threading with Tactile Perception and Reinforcement Learning](https://mlsanity.com/paper/2311.02396/) (2023)
- [FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills](https://mlsanity.com/paper/2404.19217/) (2024)
- [In-Hand Following of Deformable Linear Objects Using Dexterous Fingers with Tactile Sensing](https://mlsanity.com/paper/2403.12676/) (2024)
- [Interactive Navigation for Legged Manipulators with Learned Arm-Pushing Controller](https://mlsanity.com/paper/2503.01474/) (2025)

## Abstract (arXiv)

Deformable objects often appear in unstructured configurations. Tracing deformable objects helps bringing them into extended states and facilitating the downstream manipulation tasks. Due to the requirements for object-specific modeling or sim-to-real transfer, existing tracing methods either lack generalizability across different categories of deformable objects or struggle to complete tasks reliably in the real world. To address this, we propose a novel visual-tactile imitation learning method to achieve one-dimensional (1D) and two-dimensional (2D) deformable object tracing with a unified model. Our method is designed from both local and global perspectives based on visual and tactile sensing. Locally, we introduce a weighted loss that emphasizes actions maintaining contact near the center of the tactile image, improving fine-grained adjustment. Globally, we propose a tracing task loss that helps the policy to regulate task progression. On the hardware side, to compensate for the limited features extracted from visual information, we integrate tactile sensing into a low-cost teleoperation system considering both the teleoperator and the robot. Extensive ablation and comparative experiments on diverse 1D and 2D deformable objects demonstrate the effectiveness of our approach, achieving an average success rate of 80% on seen objects and 65% on unseen objects.