mlsanityData to 14 Oct 2026

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

arXiv 1810.10191 ↗First posted 24 Oct 2018cs.RO cs.AI cs.LGcited by 154

Not on the map. Why?

Abstract (arXiv)

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. Results for simulated and real robot experiments are presented.

Key facts. Cited by 154 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 27; 2022: 26; 2023: 21; 2024: 23; 2025: 27; 2026: 29.

154 papers.

Sort:

By area:

  1. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  3. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  4. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  5. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  6. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 146
  7. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 137
  8. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  9. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 87
  10. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  11. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  12. 20243D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensingcited by 72
38 more
  1. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 68
  2. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 68
  3. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  4. 2024ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterizationcited by 61
  5. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 59
  6. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  7. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 56
  8. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 56
  9. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  10. 2022Visuo-Tactile Transformers for Manipulationcited by 47
  11. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 41
  12. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  13. 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
  14. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 34
  15. 2022High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learningcited by 31
  16. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 31
  17. 2025Understanding the Emergence of Multimodal Representation Alignmentcited by 31
  18. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  19. 2023The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learningcited by 26
  20. 2022SEIL: Simulation-augmented Equivariant Imitation Learningcited by 23
  21. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  22. 2022SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Renderingcited by 22
  23. 2023Category-Level Multi-Part Multi-Joint 3D Shape Assemblycited by 22
  24. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 21
  25. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 20
  26. 2021Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Taskscited by 18
  27. 2021InsertionNet -- A Scalable Solution for Insertioncited by 16
  28. 2021Composable Energy Policies for Reactive Motion Generation and Reinforcement Learningcited by 16
  29. 2025A Taxonomy for Evaluating Generalist Robot Manipulation Policiescited by 16
  30. 2022Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feedingcited by 15
  31. 2022Imitation Learning for Nonprehensile Manipulation through Self-Supervised Learning Considering Motion Speedcited by 14
  32. 2024Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulationcited by 14
  33. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulationcited by 14
  34. 2021Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learningcited by 13
  35. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 13
  36. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  37. 2025Multi-Modal Manipulation via Multi-Modal Policy Consensuscited by 13
  38. 2022Visuo-Haptic Object Perception for Robots: An Overviewcited by 12

50 most cited of 154 All 154 (JSON)