# You've Got to Feel It To Believe It: Multi-Modal Bayesian Inference for Semantic and Property Prediction

arXiv: https://arxiv.org/abs/2402.05872 · HTML: https://mlsanity.com/paper/2402.05872/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2024.

## Built on by (most cited first, 7 of 7)

- [PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation](https://mlsanity.com/paper/2511.18570/) (2025)
- [LatentBKI: Open-Dictionary Continuous Mapping in Visual-Language Latent Spaces with Quantifiable Uncertainty](https://mlsanity.com/paper/2410.11783/) (2024)
- [Neural Fidelity Calibration for Informative Sim-to-Real Adaptation](https://mlsanity.com/paper/2504.08604/) (2025)
- [SLIM-VDB: A Real-Time 3D Probabilistic Semantic Mapping Framework](https://mlsanity.com/paper/2512.12945/) (2025)
- [Cross-Attentive Multiview Fusion of Vision-Language Embeddings](https://mlsanity.com/paper/2604.12551/) (2026)
- [Learning What Matters: Adaptive Information-Theoretic Objectives for Robot Exploration](https://mlsanity.com/paper/2605.12084/) (2026)
- [WCM: World-Cognition Model for Generalizable Human-Robot Interaction](https://mlsanity.com/paper/2607.22999/) (2026)

## Abstract (arXiv)

Robots must be able to understand their surroundings to perform complex tasks in challenging environments and many of these complex tasks require estimates of physical properties such as friction or weight. Estimating such properties using learning is challenging due to the large amounts of labelled data required for training and the difficulty of updating these learned models online at run time. To overcome these challenges, this paper introduces a novel, multi-modal approach for representing semantic predictions and physical property estimates jointly in a probabilistic manner. By using conjugate pairs, the proposed method enables closed-form Bayesian updates given visual and tactile measurements without requiring additional training data. The efficacy of the proposed algorithm is demonstrated through several hardware experiments. In particular, this paper illustrates that by conditioning semantic classifications on physical properties, the proposed method quantitatively outperforms state-of-the-art semantic classification methods that rely on vision alone. To further illustrate its utility, the proposed method is used in several applications including to represent affordance-based properties probabilistically and a challenging terrain traversal task using a legged robot. In the latter task, the proposed method represents the coefficient of friction of the terrain probabilistically, which enables the use of an on-line risk-aware planner that switches the legged robot from a dynamic gait to a static, stable gait when the expected value of the coefficient of friction falls below a given threshold. Videos of these case studies as well as the open-source C++ and ROS interface can be found at https://roahmlab.github.io/multimodal_mapping/.