# Multimodal VAE Active Inference Controller

arXiv: https://arxiv.org/abs/2103.04412 · HTML: https://mlsanity.com/paper/2103.04412/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 7 Mar 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [State Representation Learning for Control: An Overview](https://mlsanity.com/paper/1802.04181/) (2018)
- [The free energy principle for action and perception: A mathematical review](https://mlsanity.com/paper/1705.09156/) (2017)
- [On the Relationship Between Active Inference and Control as Inference](https://mlsanity.com/paper/2006.12964/) (2020)
- [End-to-End Pixel-Based Deep Active Inference for Body Perception and Action](https://mlsanity.com/paper/2001.05847/) (2019)
- [Deep Active Inference for Partially Observable MDPs](https://mlsanity.com/paper/2009.03622/) (2020)
- [Adaptive robot body learning and estimation through predictive coding](https://mlsanity.com/paper/1805.03104/) (2018)
- [A Novel Adaptive Controller for Robot Manipulators based on Active Inference](https://mlsanity.com/paper/1909.12768/) (2019)
- [Robot self/other distinction: active inference meets neural networks learning in a mirror](https://mlsanity.com/paper/2004.05473/) (2020)
- [Active Inference for Integrated State-Estimation, Control, and Learning](https://mlsanity.com/paper/2005.05894/) (2020)
- [A deep active inference model of the rubber-hand illusion](https://mlsanity.com/paper/2008.07408/) (2020)
- [Goal-Directed Behavior under Variational Predictive Coding: Dynamic Organization of Visual Attention and Working Memory](https://mlsanity.com/paper/1903.04932/) (2019)

## Built on by (most cited first, 21 of 21)

- [Active Inference in Robotics and Artificial Agents: Survey and Challenges](https://mlsanity.com/paper/2112.01871/) (2021)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [Geometric Multimodal Contrastive Representation Learning](https://mlsanity.com/paper/2202.03390/) (2022)
- [World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challenges](https://mlsanity.com/paper/2301.05832/) (2023)
- [Mamba as a motion encoder for robotic imitation learning](https://mlsanity.com/paper/2409.02636/) (2024)
- [Adaptation through prediction: multisensory active inference torque control](https://mlsanity.com/paper/2112.06752/) (2021)
- [Neuroscience-inspired perception-action in robotics: applying active inference for state estimation, control and self-perception](https://mlsanity.com/paper/2105.04261/) (2021)
- [Deep Active Inference for Pixel-Based Discrete Control: Evaluation on the Car Racing Problem](https://mlsanity.com/paper/2109.04155/) (2021)
- [Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization](https://mlsanity.com/paper/2109.11045/) (2021)
- [Generative Machine Learning in Adaptive Control of Dynamic Manufacturing Processes: A Review](https://mlsanity.com/paper/2505.00210/) (2025)
- [Robot Localization and Navigation through Predictive Processing using LiDAR](https://mlsanity.com/paper/2109.04139/) (2021)
- [TC-VAE: Uncovering Out-of-Distribution Data Generative Factors](https://mlsanity.com/paper/2304.04103/) (2023)
- [Analyzing Multimodal Integration in the Variational Autoencoder from an Information-Theoretic Perspective](https://mlsanity.com/paper/2411.00522/) (2024)
- [MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning](https://mlsanity.com/paper/2607.16789/) (2026)
- [Learn Proportional Derivative Controllable Latent Space from Pixels](https://mlsanity.com/paper/2110.08239/) (2021)
- [Dynamic planning in hierarchical active inference](https://mlsanity.com/paper/2402.11658/) (2024)
- [Seeing Through Uncertainty: Free-Energy-Inspired Real-Time Adaptation for Robust Visual Navigation](https://mlsanity.com/paper/2403.01977/) (2024)
- [Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks](https://mlsanity.com/paper/2404.01932/) (2024)
- [Active Inference for an Intelligent Agent in Autonomous Reconnaissance Missions](https://mlsanity.com/paper/2510.17450/) (2025)
- [Bayesian Inverse Games with High-Dimensional Multi-Modal Observations](https://mlsanity.com/paper/2601.00696/) (2026)
- [Exploring Temporal Representation in Neural Processes for Multimodal Action Prediction](https://mlsanity.com/paper/2604.08418/) (2026)

## Abstract (arXiv)

Active inference, a theoretical construct inspired by brain processing, is a promising alternative to control artificial agents. However, current methods do not yet scale to high-dimensional inputs in continuous control. Here we present a novel active inference torque controller for industrial arms that maintains the adaptive characteristics of previous proprioceptive approaches but also enables large-scale multimodal integration (e.g., raw images). We extended our previous mathematical formulation by including multimodal state representation learning using a linearly coupled multimodal variational autoencoder. We evaluated our model on a simulated 7DOF Franka Emika Panda robot arm and compared its behavior with a previous active inference baseline and the Panda built-in optimized controller. Results showed improved tracking and control in goal-directed reaching due to the increased representation power, high robustness to noise and adaptability in changes on the environmental conditions and robot parameters without the need to relearn the generative models nor parameters retuning.