mlsanityData to 14 Oct 2026

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

arXiv 1907.13098 ↗First posted 28 Jul 2019cs.RO cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is non-trivial to manually design a robot controller that combines these modalities which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. In this work, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. Evaluating our method on a peg insertion task, we show that it generalizes over varying geometries, configurations, and clearances, while being robust to external perturbations. We also systematically study different self-supervised learning objectives and representation learning architectures. Results are presented in simulation and on a physical robot.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 16; 2023: 14; 2024: 17; 2025: 16; 2026: 12.

84 papers.

Sort:

By area:

  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  2. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 183
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  4. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  5. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  6. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 90
  7. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 84
  8. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 84
  9. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 83
  10. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 79
  11. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 68
  12. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 62
38 more
  1. 2022Factory: Fast Contact for Robotic Assemblycited by 54
  2. 2022Visuo-Tactile Transformers for Manipulationcited by 47
  3. 2024What to align in multimodal contrastive learning?cited by 47
  4. 2025TLA: Tactile-Language-Action Model for Contact-Rich Manipulationcited by 47
  5. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 42
  6. 2025OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensingcited by 38
  7. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 35
  8. 2022DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanationscited by 34
  9. 2022High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learningcited by 31
  10. 2023On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfercited by 26
  11. 2022Visuomotor Control in Multi-Object Scenes Using Object-Aware Representationscited by 21
  12. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 20
  13. 2023Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challengescited by 18
  14. 2024SRL-VIC: A Variable Stiffness-Based Safe Reinforcement Learning for Contact-Rich Robotic Taskscited by 15
  15. 2022Bayesian Imitation Learning for End-to-End Mobile Manipulationcited by 14
  16. 2021Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learningcited by 13
  17. 2021Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Modelscited by 12
  18. 2023Learning Unseen Modality Interactioncited by 12
  19. 2024Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skinscited by 12
  20. 2024DISCO: Efficient Diffusion Solver for Large-Scale Combinatorial Optimization Problemscited by 11
  21. 2021Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertaintycited by 10
  22. 2022RCA: Ride Comfort-Aware Visual Navigation via Self-Supervised Learningcited by 10
  23. 2023Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspectivecited by 10
  24. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Modelscited by 10
  25. 2025End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collectioncited by 9
  26. 2024M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulationcited by 8
  27. 2025InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactionscited by 7
  28. 2021Imitation and Supervised Learning of Compliance for Robotic Assemblycited by 6
  29. 2022Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrationscited by 6
  30. 2023Identifying Important Sensory Feedback for Learning Locomotion Skillscited by 6
  31. 2024Built Different: Tactile Perception to Overcome Cross-Embodiment Capability Differences in Collaborative Manipulationcited by 6
  32. 2025CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systemscited by 6
  33. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  34. 2026ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learningcited by 5
  35. 2024Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensorscited by 4
  36. 2022Leveraging Haptic Feedback to Improve Data Quality and Quantity for Deep Imitation Learning Modelscited by 3
  37. 2022Multimodal Learning for Multi-Omics: A Surveycited by 3
  38. 2023Bayesian and Neural Inference on LSTM-based Object Recognition from Tactile and Kinesthetic Informationcited by 3

50 most cited of 84 All 84 (JSON)