mlsanityData to 14 Oct 2026

See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation

arXiv 2212.03858 ↗First posted 7 Dec 2022cs.RO cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two modalities, such as vision and touch. In this work, we systematically study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor, with all three sensory modalities fused with a self-attention model. Results on two challenging tasks, dense packing and pouring, demonstrate the necessity and power of multisensory perception for robotic manipulation: vision displays the global status of the robot but can often suffer from occlusion, audio provides immediate feedback of key moments that are even invisible, and touch offers precise local geometry for decision making. Leveraging all three modalities, our robotic system significantly outperforms prior methods.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 45 of its 68 references that are in the map. First posted on arXiv 7 Dec 2022.

Builds on

45 of 68 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2018Learning Dexterous In-Hand Manipulationcited by 687
  12. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
33 more
  1. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  2. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  3. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  4. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  5. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  6. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  7. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  8. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 154
  9. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  10. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 142
  11. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 102
  12. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 84
  13. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  14. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 72
  15. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 68
  16. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 67
  17. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  18. 2021Tactile-RL for Insertion: Generalization to Objects of Unknown Geometrycited by 48
  19. 20203D Shape Reconstruction from Vision and Touchcited by 42
  20. 2016The Curious Robot: Learning Visual Representations via Physical Interactionscited by 38
  21. 2022Audio-Adaptive Activity Recognition Across Video Domainscited by 35
  22. 2021GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Fingercited by 34
  23. 2021Active 3D Shape Reconstruction from Vision and Touchcited by 34
  24. 2021ShapeMap 3-D: Efficient shape mapping through dense touch and visioncited by 31
  25. 2020Multimodal Sensor Fusion with Differentiable Filterscited by 29
  26. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  27. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  28. 2020Tactile Object Pose Estimation from the First Touch with Geometric Contact Renderingcited by 20
  29. 2020Swoosh! Rattle! Thump! -- Actions that Soundcited by 15
  30. 2019Tactile-Based Insertion for Dense Box-Packingcited by 12
  31. 2020STReSSD: Sim-To-Real from Sound for Stochastic Dynamicscited by 12
  32. 2019Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouringcited by 11
  33. 2021Playing with Food: Learning Food Item Representations through Interactive Explorationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 16; 2025: 23; 2026: 21.

68 papers.

Sort:

By area:

  1. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  2. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 60
  3. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 48
  4. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 45
  5. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 42
  6. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 41
  7. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  8. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 34
  9. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Controlcited by 33
  10. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 31
  11. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 31
  12. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
38 more
  1. 2023The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learningcited by 26
  2. 2024AnySkin: Plug-and-play Skin Sensing for Robotic Touchcited by 23
  3. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 20
  4. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 20
  5. 2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 19
  6. 2025Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulationcited by 18
  7. 2024NeRFs in Robotics: A Surveycited by 16
  8. 2024Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulationcited by 14
  9. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  10. 2025Multi-Modal Manipulation via Multi-Modal Policy Consensuscited by 13
  11. 2024Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skinscited by 12
  12. 2024Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learningcited by 11
  13. 2025Robotic Compliant Object Prying Using Diffusion Policy Guided by Vision and Force Observationscited by 10
  14. 2023Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensorcited by 9
  15. 2024M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulationcited by 8
  16. 2025Universal Visuo-Tactile Video Understanding for Embodied Interactioncited by 8
  17. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 6
  18. 2026ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learningcited by 5
  19. 2026UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Modelscited by 5
  20. 2023MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learningcited by 4
  21. 2024Ensuring Force Safety in Vision-Guided Robotic Manipulation via Implicit Tactile Calibrationcited by 4
  22. 2025On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lightingcited by 4
  23. 2025DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulationcited by 4
  24. 2025Multimodal Diffusion Forcing for Forceful Manipulationcited by 4
  25. 2025Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learningcited by 3
  26. 2024Multi-modal perception for soft robotic interactions using generative modelscited by 2
  27. 2025RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Datacited by 2
  28. 2025Force-Modulated Visual Policy for Robot-Assisted Dressing with Arm Motionscited by 2
  29. 2026Audio-Visual Intelligence in Large Foundation Modelscited by 2
  30. 2026Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlationcited by 2
  31. 2026TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulationcited by 2
  32. 2025M2R2: MultiModal Robotic Representation for Temporal Action Segmentationcited by 1
  33. 2025RETRO: REthinking Tactile Representation Learning with Material PriOrscited by 1
  34. 2025Gentle Object Retraction in Dense Clutter Using Multimodal Force Sensing and Imitation Learningcited by 1
  35. 2025Grasp Like Humans: Learning Generalizable Multi-Fingered Grasping from Human Proprioceptive Sensorimotor Integrationcited by 1
  36. 2026MambaVF: State Space Model for Efficient Video Fusioncited by 1
  37. 2026Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robotscited by 1
  38. 2026Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusioncited by 1

50 most cited of 68 All 68 (JSON)