# Cross-Modal Visuo-Tactile Object Perception

arXiv: https://arxiv.org/abs/2604.02108 · HTML: https://mlsanity.com/paper/2604.02108/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 52 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1907.13098/) (2019)
- [ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation](https://mlsanity.com/paper/2506.15953/) (2025)
- [Learning Flat Latent Manifolds with VAEs](https://mlsanity.com/paper/2002.04881/) (2020)
- [Latent Matters: Learning Deep State-Space Models](https://mlsanity.com/paper/2602.23050/) (2026)
- [Conductance-based dendrites perform Bayes-optimal cue integration](https://mlsanity.com/paper/2104.13238/) (2021)
- [Visuo-Tactile based Predictive Cross Modal Perception for Object Exploration in Robotics](https://mlsanity.com/paper/2405.12634/) (2024)
- [Perception, Control and Hardware for In-Hand Slip-Aware Object Manipulation with Parallel Grippers](https://mlsanity.com/paper/2410.19660/) (2024)
- [Predictive Visuo-Tactile Interactive Perception Framework for Object Properties Inference](https://mlsanity.com/paper/2411.09020/) (2024)

## Abstract (arXiv)

Estimating physical properties is critical for safe and efficient autonomous robotic manipulation, particularly during contact-rich interactions. In such settings, vision and tactile sensing provide complementary information about object geometry, pose, inertia, stiffness, and contact dynamics, such as stick-slip behavior. However, these properties are only indirectly observable and cannot always be modeled precisely (e.g., deformation in non-rigid objects coupled with nonlinear contact friction), making the estimation problem inherently complex and requiring sustained exploitation of visuo-tactile sensory information during action. Existing visuo-tactile perception frameworks have primarily emphasized forceful sensor fusion or static cross-modal alignment, with limited consideration of how uncertainty and beliefs about object properties evolve over time. Inspired by human multi-sensory perception and active inference, we propose the Cross-Modal Latent Filter (CMLF) to learn a structured, causal latent state-space of physical object properties. CMLF supports bidirectional transfer of cross-modal priors between vision and touch and integrates sensory evidence through a Bayesian inference process that evolves over time. Real-world robotic experiments demonstrate that CMLF improves the efficiency and robustness of latent physical properties estimation under uncertainty compared to baseline approaches. Beyond performance gains, the model exhibits perceptual coupling phenomena analogous to those observed in humans, including susceptibility to cross-modal illusions and similar trajectories in learning cross-sensory associations. Together, these results constitutes a significant step toward generalizable, robust and physically consistent cross-modal integration for robotic multi-sensory perception.