# ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations

arXiv: https://arxiv.org/abs/2506.20757 · HTML: https://mlsanity.com/paper/2506.20757/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 25 Jun 2025.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?](https://mlsanity.com/paper/1710.05512/) (2017)
- [Visuo-Tactile Transformers for Manipulation](https://mlsanity.com/paper/2210.00121/) (2022)
- [Contrastive Multimodal Fusion with TupleInfoNCE](https://mlsanity.com/paper/2107.02575/) (2021)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features](https://mlsanity.com/paper/2209.13042/) (2022)
- [Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training](https://mlsanity.com/paper/2401.12024/) (2024)
- [ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition](https://mlsanity.com/paper/1802.07490/) (2018)
- [Robotic Tactile Perception of Object Properties: A Review](https://mlsanity.com/paper/1711.03810/) (2017)
- [Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion](https://mlsanity.com/paper/2108.05009/) (2021)
- [GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation](https://mlsanity.com/paper/2008.05404/) (2020)
- [S3M-Net: Joint Learning of Semantic Segmentation and Stereo Matching for Autonomous Driving](https://mlsanity.com/paper/2401.11414/) (2024)
- [Spatio-temporal Attention Model for Tactile Texture Recognition](https://mlsanity.com/paper/2008.04442/) (2020)
- [Multimodal Zero-Shot Learning for Tactile Texture Recognition](https://mlsanity.com/paper/2306.12705/) (2023)
- [TransForce: Transferable Force Prediction for Vision-based Tactile Sensors with Sequential Image Translation](https://mlsanity.com/paper/2409.09870/) (2024)
- [TouchRoller: A Rolling Optical Tactile Sensor for Rapid Assessment of Large Surfaces](https://mlsanity.com/paper/2103.00595/) (2021)
- [Deep Domain Adaptation Regression for Force Calibration of Optical Tactile Sensors](https://mlsanity.com/paper/2407.14380/) (2024)
- [CDI3D: Cross-guided Dense-view Interpolation for 3D Reconstruction](https://mlsanity.com/paper/2503.08005/) (2025)

## Built on by (most cited first, 8 of 8)

- [ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learning](https://mlsanity.com/paper/2602.11643/) (2026)
- [DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter](https://mlsanity.com/paper/2602.05513/) (2026)
- [ViTacGen: Robotic Pushing with Vision-to-Touch Generation](https://mlsanity.com/paper/2510.14117/) (2025)
- [Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms](https://mlsanity.com/paper/2605.17336/) (2026)
- [Symmetry-Aware Fusion of Vision and Tactile Sensing via Bilateral Force Priors for Robotic Manipulation](https://mlsanity.com/paper/2602.13689/) (2026)
- [ViTac-Tracing: Visual-Tactile Imitation Learning of Deformable Object Tracing](https://mlsanity.com/paper/2603.18784/) (2026)
- [VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds](https://mlsanity.com/paper/2607.16146/) (2026)
- [Language-Guided Representation Learning for Robust Cross-Sensor Material Recognition](https://mlsanity.com/paper/2609.14783/) (2026)

## Abstract (arXiv)

Vision and touch are two fundamental sensory modalities for robots, offering complementary information that enhances perception and manipulation tasks. Previous research has attempted to jointly learn visual-tactile representations to extract more meaningful information. However, these approaches often rely on direct combination, such as feature addition and concatenation, for modality fusion, which tend to result in poor feature integration. In this paper, we propose ConViTac, a visual-tactile representation learning network designed to enhance the alignment of features during fusion using contrastive representations. Our key contribution is a Contrastive Embedding Conditioning (CEC) mechanism that leverages a contrastive encoder pretrained through self-supervised contrastive learning to project visual and tactile inputs into unified latent embeddings. These embeddings are used to couple visual-tactile feature fusion through cross-modal attention, aiming at aligning the unified representations and enhancing performance on downstream tasks. We conduct extensive experiments to demonstrate the superiority of ConViTac in real world over current state-of-the-art methods and the effectiveness of our proposed CEC mechanism, which improves accuracy by up to 12.0% in material classification and grasping prediction tasks.