# Multi-modal perception for soft robotic interactions using generative models

arXiv: https://arxiv.org/abs/2404.04220 · HTML: https://mlsanity.com/paper/2404.04220/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Revisiting Active Perception](https://mlsanity.com/paper/1603.02729/) (2016)
- [Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1907.13098/) (2019)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019)
- [Soft Robots Modeling: a Structured Overview](https://mlsanity.com/paper/2112.03645/) (2021)
- ["Touching to See" and "Seeing to Feel": Robotic Cross-modal SensoryData Generation for Visual-Tactile Perception](https://mlsanity.com/paper/1902.06273/) (2019)
- [Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors](https://mlsanity.com/paper/2012.00201/) (2020)
- [Multimodel Sensor Fusion for Learning Rich Models for Interacting Soft Robots](https://mlsanity.com/paper/2205.04202/) (2022)
- [Plant-inspired behavior-based controller to enable reaching in redundant continuum robot arms](https://mlsanity.com/paper/2304.07188/) (2023)

## Built on by (most cited first, 2 of 2)

- [Continual Learning for Multimodal Data Fusion of a Soft Gripper](https://mlsanity.com/paper/2409.13792/) (2024)
- [Towards Interpretable Visuo-Tactile Predictive Models for Soft Robot Interactions](https://mlsanity.com/paper/2407.12197/) (2024)

## Abstract (arXiv)

Perception is essential for the active interaction of physical agents with the external environment. The integration of multiple sensory modalities, such as touch and vision, enhances this perceptual process, creating a more comprehensive and robust understanding of the world. Such fusion is particularly useful for highly deformable bodies such as soft robots. Developing a compact, yet comprehensive state representation from multi-sensory inputs can pave the way for the development of complex control strategies. This paper introduces a perception model that harmonizes data from diverse modalities to build a holistic state representation and assimilate essential information. The model relies on the causality between sensory input and robotic actions, employing a generative model to efficiently compress fused information and predict the next observation. We present, for the first time, a study on how touch can be predicted from vision and proprioception on soft robots, the importance of the cross-modal generation and why this is essential for soft robotic interactions in unstructured environments.