mlsanityData to 7 Oct 2026

Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor

arXiv 2311.01248 ↗First posted 2 Nov 2023cs.RO cs.AI cs.LGcited by 9

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Contact-rich tasks continue to present many challenges for robotic manipulation. In this work, we leverage a multimodal visuotactile sensor within the framework of imitation learning (IL) to perform contact-rich tasks that involve relative motion (e.g., slipping and sliding) between the end-effector and the manipulated object. We introduce two algorithmic contributions, tactile force matching and learned mode switching, as complimentary methods for improving IL. Tactile force matching enhances kinesthetic teaching by reading approximate forces during the demonstration and generating an adapted robot trajectory that recreates the recorded forces. Learned mode switching uses IL to couple visual and tactile sensor modes with the learned motion policy, simplifying the transition from reaching to contacting. We perform robotic manipulation experiments on four door-opening tasks with a variety of observation and algorithm configurations to study the utility of multimodal visuotactile sensing and our proposed improvements. Our results show that the inclusion of force matching raises average policy success rates by 62.5%, visuotactile mode switching by 30.3%, and visuotactile data as a policy input by 42.5%, emphasizing the value of see-through tactile sensing for IL, both for data collection to allow force matching, and for policy execution to enable accurate task feedback. Project site: https://papers.starslab.ca/sts-il/

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 53 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

22 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Immersive Demonstrations are the Key to Imitation Learning22 Jan

Other references · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  4. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  5. 2019Causal Confusion in Imitation Learningcited by 286
  6. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  7. 2021What Matters for Adversarial Imitation Learning?cited by 70
  8. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  9. 2019Cable Manipulation with a Tactile-Reactive Grippercited by 44
  10. 2020OmniTact: A Multi-Directional High Resolution Touch Sensorcited by 27
  11. 2022RB2: Robotic Manipulation Benchmarking with a Twistcited by 23
  12. 2018Dense Tactile Force Distribution Estimation using GelSlim and inverse FEMcited by 17
9 more
  1. 2022Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touchcited by 17
  2. 2022Temporal Logic Imitation: Learning Plan-Satisficing Motion Policies from Demonstrationscited by 10
  3. 2022Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Taskscited by 10
  4. 2020Seeing Through your Skin: Recognizing Objects with a Novel Visuotactile Sensorcited by 8
  5. 2023Imitation Learning from Observation through Optimal Transportcited by 7
  6. 2020Heteroscedastic Uncertainty for Robust Generative Latent Dynamicscited by 5
  7. 2021Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrationscited by 5
  8. 2022Learning Sequential Latent Variable Models from Multimodal Time Series Datacited by 4
  9. 2018Self-Calibration of Mobile Manipulator Kinematic and Sensor Extrinsic Parameters Through Contact-Based Interactioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  2. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  3. 2025Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulationcited by 6
  4. 2025On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lightingcited by 4
  5. 2025UniConFlow: A Unified Constrained Flow-Matching Framework for Certified Motion Planningcited by 4
  6. 2026Modular Sensory Stream for Integrating Physical Feedback in Vision-Language-Action Modelscited by 2
  7. 2025Gentle Object Retraction in Dense Clutter Using Multimodal Force Sensing and Imitation Learningcited by 1
  8. 2026Force Generative Imitation Learning: Bridging Position Trajectory and Force Commands through Control Techniquecited by 0
  9. 2026S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Informationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Visuo-Tactile Data Collection System with Haptic Feedback for Coarse-to-Fine Imitation Learning
  2. 2026Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation ↗
  3. 2026TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
  4. 2025Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation
  5. 2025Towards Forceful Robotic Foundation Models: a Literature Survey
  6. 2026UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
  7. 2026ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
  8. 2025GelFusion: Enhancing Robotic Manipulation under Visual Constraints via Visuotactile Fusion