# Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning

arXiv: https://arxiv.org/abs/2204.00117 · HTML: https://mlsanity.com/paper/2204.00117/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects](https://mlsanity.com/paper/1911.06283/) (2019)
- [ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition](https://mlsanity.com/paper/1802.07490/) (2018)
- [Robotic Tactile Perception of Object Properties: A Review](https://mlsanity.com/paper/1711.03810/) (2017)
- [Connecting Look and Feel: Associating the visual and tactile properties of physical materials](https://mlsanity.com/paper/1704.03822/) (2017)
- [GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation](https://mlsanity.com/paper/2008.05404/) (2020)
- [Generation of GelSight Tactile Images for Sim2Real Learning](https://mlsanity.com/paper/2101.07169/) (2021)
- [Surface Following using Deep Reinforcement Learning and a GelSightTactile Sensor](https://mlsanity.com/paper/1912.00745/) (2019)

## Built on by (most cited first, 9 of 9)

- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning](https://mlsanity.com/paper/2311.00924/) (2023)
- [AcTExplore: Active Tactile Exploration of Unknown Objects](https://mlsanity.com/paper/2310.08745/) (2023)
- [Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs](https://mlsanity.com/paper/2504.15595/) (2025)
- [Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control](https://mlsanity.com/paper/2307.14510/) (2023)
- [Image-to-Force Estimation for Soft Tissue Interaction in Robotic-Assisted Surgery Using Structured Light](https://mlsanity.com/paper/2501.08593/) (2025)
- [RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data](https://mlsanity.com/paper/2505.14270/) (2025)
- [URPlanner: A Universal Paradigm For Collision-Free Robotic Motion Planning Based on Deep Reinforcement Learning](https://mlsanity.com/paper/2505.20175/) (2025)
- [ViTac-Tracing: Visual-Tactile Imitation Learning of Deformable Object Tracing](https://mlsanity.com/paper/2603.18784/) (2026)

## Abstract (arXiv)

Manipulation of deformable objects is a challenging task for a robot. It will be problematic to use a single sensory input to track the behaviour of such objects: vision can be subjected to occlusions, whereas tactile inputs cannot capture the global information that is useful for the task. In this paper, we study the problem of using vision and tactile inputs together to complete the task of following deformable linear objects, for the first time. We create a Reinforcement Learning agent using different sensing modalities and investigate how its behaviour can be boosted using visual-tactile fusion, compared to using a single sensing modality. To this end, we developed a benchmark in simulation for manipulating the deformable linear objects using multimodal sensing inputs. The policy of the agent uses distilled information, e.g., the pose of the object in both visual and tactile perspectives, instead of the raw sensing signals, so that it can be directly transferred to real environments. In this way, we disentangle the perception system and the learned control policy. Our extensive experiments show that the use of both vision and tactile inputs, together with proprioception, allows the agent to complete the task in up to 92% of cases, compared to 77% when only one of the signals is given. Our results can provide valuable insights for the future design of tactile sensors and for deformable objects manipulation.