# Towards Interpretable Visuo-Tactile Predictive Models for Soft Robot Interactions

arXiv: https://arxiv.org/abs/2407.12197 · HTML: https://mlsanity.com/paper/2407.12197/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 9 of 9)

- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [BERTology Meets Biology: Interpreting Attention in Protein Language Models](https://mlsanity.com/paper/2006.15222/) (2020)
- [Multimodal Sensor Fusion with Differentiable Filters](https://mlsanity.com/paper/2010.13021/) (2020)
- [An overview of differentiable particle filters for data-adaptive sequential Bayesian inference](https://mlsanity.com/paper/2302.09639/) (2023)
- [PCAAE: Principal Component Analysis Autoencoder for organising the latent space of generative networks](https://mlsanity.com/paper/2006.07827/) (2020)
- [Multimodal sensor fusion in the latent representation space](https://mlsanity.com/paper/2208.02183/) (2022)
- [Bodily aware soft robots: integration of proprioceptive and exteroceptive sensors](https://mlsanity.com/paper/1710.05419/) (2017)
- [Multi-modal perception for soft robotic interactions using generative models](https://mlsanity.com/paper/2404.04220/) (2024)

## Abstract (arXiv)

Autonomous systems face the intricate challenge of navigating unpredictable environments and interacting with external objects. The successful integration of robotic agents into real-world situations hinges on their perception capabilities, which involve amalgamating world models and predictive skills. Effective perception models build upon the fusion of various sensory modalities to probe the surroundings. Deep learning applied to raw sensory modalities offers a viable option. However, learning-based perceptive representations become difficult to interpret. This challenge is particularly pronounced in soft robots, where the compliance of structures and materials makes prediction even harder. Our work addresses this complexity by harnessing a generative model to construct a multi-modal perception model for soft robots and to leverage proprioceptive and visual information to anticipate and interpret contact interactions with external objects. A suite of tools to interpret the perception model is furnished, shedding light on the fusion and prediction processes across multiple sensory inputs after the learning phase. We will delve into the outlooks of the perception model and its implications for control purposes.