# Learning Intuitive Physics with Multimodal Generative Models

arXiv: https://arxiv.org/abs/2101.04454 · HTML: https://mlsanity.com/paper/2101.04454/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 12 Jan 2021.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018)
- [Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation](https://mlsanity.com/paper/1909.05829/) (2019)
- [Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions](https://mlsanity.com/paper/1410.7827/) (2014)
- [The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?](https://mlsanity.com/paper/1710.05512/) (2017)
- [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [Goal-Aware Prediction: Learning to Model What Matters](https://mlsanity.com/paper/2007.07170/) (2020)
- [3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2007.03778/) (2020)
- [Time-Agnostic Prediction: Predicting Predictable Video Frames](https://mlsanity.com/paper/1808.07784/) (2018)
- [3D Shape Perception from Monocular Vision, Touch, and Shape Priors](https://mlsanity.com/paper/1808.03247/) (2018)
- [OmniTact: A Multi-Directional High Resolution Touch Sensor](https://mlsanity.com/paper/2003.06965/) (2020)
- [ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition](https://mlsanity.com/paper/1802.07490/) (2018)
- [Connecting Look and Feel: Associating the visual and tactile properties of physical materials](https://mlsanity.com/paper/1704.03822/) (2017)
- [Variational Encoding of Complex Dynamics](https://mlsanity.com/paper/1711.08576/) (2017)
- [Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning](https://mlsanity.com/paper/1904.05869/) (2019)
- [Soft-Bubble grippers for robust and perceptive manipulation](https://mlsanity.com/paper/2004.03691/) (2020)
- ["Touching to See" and "Seeing to Feel": Robotic Cross-modal SensoryData Generation for Visual-Tactile Perception](https://mlsanity.com/paper/1902.06273/) (2019)
- [Multi-Modal Geometric Learning for Grasping and Manipulation](https://mlsanity.com/paper/1803.07671/) (2018)
- [Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models](https://mlsanity.com/paper/1808.04768/) (2018)
- [Seeing Through your Skin: Recognizing Objects with a Novel Visuotactile Sensor](https://mlsanity.com/paper/2011.09552/) (2020)
- [Multimodal dynamics modeling for off-road autonomous vehicles](https://mlsanity.com/paper/2011.11751/) (2020)

## Built on by (most cited first, 4 of 4)

- [Multimodal learning with graphs](https://mlsanity.com/paper/2209.03299/) (2022)
- [RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Rates](https://mlsanity.com/paper/2312.10386/) (2023)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [Towards deployment-centric multimodal AI beyond vision and language](https://mlsanity.com/paper/2504.03603/) (2025)

## Abstract (arXiv)

Predicting the future interaction of objects when they come into contact with their environment is key for autonomous agents to take intelligent and anticipatory actions. This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes. Visual information captures object properties such as 3D shape and location, while tactile information provides critical cues about interaction forces and resulting object motion when it makes contact with the environment. Utilizing a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces, our system captures both the visual appearance and the tactile properties of objects. We interpret the dual stream signals from the sensor using a Multimodal Variational Autoencoder (MVAE), allowing us to capture both modalities of contacting objects and to develop a mapping from visual to tactile interaction and vice-versa. Additionally, the perceptual system can be used to infer the outcome of future physical interactions, which we validate through simulated and real-world experiments in which the resting state of an object is predicted from given initial conditions.