mlsanityData to 7 Oct 2026

Visuo-Haptic Object Perception for Robots: An Overview

arXiv 2203.11544 ↗First posted 22 Mar 2022cs.RO cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

The object perception capabilities of humans are impressive, and this becomes even more evident when trying to develop solutions with a similar proficiency in autonomous robots. While there have been notable advancements in the technologies for artificial vision and touch, the effective integration of these two sensory modalities in robotic applications still needs to be improved, and several open challenges exist. Taking inspiration from how humans combine visual and haptic perception to perceive object properties and drive the execution of manual tasks, this article summarises the current state of the art of visuo-haptic object perception in robots. Firstly, the biological basis of human multimodal object perception is outlined. Then, the latest advances in sensing technologies and data collection strategies for robots are discussed. Next, an overview of the main computational techniques is presented, highlighting the main challenges of multimodal machine learning and presenting a few representative articles in the areas of robotic object recognition, peripersonal space representation and manipulation. Finally, informed by the latest advancements and open challenges, this article outlines promising new research directions.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 35 of its 187 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

35 of 187 references.

References · 35

  1. 2019Event-based Vision: A Surveycited by 1,010
  2. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  3. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  4. 2018Object Detection with Deep Learning: A Reviewcited by 524
  5. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  6. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  7. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  8. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  9. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  10. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  11. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  12. 2013Data-Driven Grasp Synthesis - A Surveycited by 104
23 more
  1. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  2. 2020Deep Visual Domain Adaptationcited by 91
  3. 2018Jacquard: A Large Scale Dataset for Robotic Grasp Detectioncited by 86
  4. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 65
  5. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 59
  6. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  7. 2020Event-Driven Visual-Tactile Sensing and Learning for Robotscited by 27
  8. 2021Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projectionscited by 27
  9. 2018ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognitioncited by 26
  10. 2021A Survey of Robot Manipulation in Contactcited by 26
  11. 2017Robotic Tactile Perception of Object Properties: A Reviewcited by 24
  12. 2018Slip Detection with Combined Tactile and Visual Informationcited by 23
  13. 2018Learning to Grasp Without Seeingcited by 19
  14. 2015Deep Learning for Tactile Understanding From Visual and Haptic Datacited by 18
  15. 2016Unscented Bayesian Optimization for Safe Robot Graspingcited by 18
  16. 2020Soft-Bubble grippers for robust and perceptive manipulationcited by 18
  17. 2019"Touching to See" and "Seeing to Feel": Robotic Cross-modal SensoryData Generation for Visual-Tactile Perceptioncited by 16
  18. 2019TactileGCN: A Graph Convolutional Network for Predicting Grasp Stability with Tactile Sensorscited by 15
  19. 2018Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformationscited by 12
  20. 2020Sim-to-Real Transfer for Optical Tactile Sensingcited by 7
  21. 2021Reducing Tactile Sim2Real Domain Gaps via Deep Texture Generation Networkscited by 5
  22. 2020A Transfer Learning Approach to Cross-Modal Object Recognition: From Visual Observation to Robotic Haptic Explorationcited by 4
  23. 2021AU Dataset for Visuo-Haptic Object Recognition for Robotscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 2.

12 papers.

  1. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  2. 2024M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulationcited by 7
  3. 2025V-HOP: Visuo-Haptic 6D Object Pose Trackingcited by 4
  4. 2024Optimizing BioTac Simulation for Realistic Tactile Perceptioncited by 2
  5. 2024FusionSense: Bridging Common Sense, Vision, and Touch for Robust Sparse-View Reconstructioncited by 2
  6. 2024Transferring Tactile Data Across Sensorscited by 2
  7. 2024ACROSS: A Deformation-Based Cross-Modal Representation for Robotic Tactile Perceptioncited by 2
  8. 2024Multi-Scenario Reasoning: Unlocking Cognitive Autonomy in Humanoid Robots for Multimodal Understandingcited by 1
  9. 2025Task-Optimized Convolutional Recurrent Networks Align with Tactile Processing in the Rodent Braincited by 1
  10. 2025Advances in Compliance Detection: Novel Models Using Vision-Based Tactile Sensorscited by 0
  11. 2026Learning to Weigh Waste: A Physics-Informed Multimodal Fusion Framework and Large-Scale Dataset for Commercial and Industrial Applicationscited by 0
  12. 2026SPLIT: Separating Physical-Contact via Latent Arithmetic in Image-Based Tactile Sensorscited by 0