# Variational Inference MPC for Bayesian Model-based Reinforcement Learning

arXiv: https://arxiv.org/abs/1907.04202 · HTML: https://mlsanity.com/paper/1907.04202/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2019.

## Built on by (most cited first, 30 of 39)

- [Active Inference in Robotics and Artificial Agents: Survey and Challenges](https://mlsanity.com/paper/2112.01871/) (2021)
- [Generalized Schr"odinger Bridge Matching](https://mlsanity.com/paper/2310.02233/) (2023)
- [DreamingV2: Reinforcement Learning with Discrete World Models without Reconstruction](https://mlsanity.com/paper/2203.00494/) (2022)
- [Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models](https://mlsanity.com/paper/2301.04741/) (2023)
- [Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.17258/) (2021)
- [Variational Inference MPC using Normalizing Flows and Out-of-Distribution Projection](https://mlsanity.com/paper/2205.04667/) (2022)
- [Variational Inference MPC using Tsallis Divergence](https://mlsanity.com/paper/2104.00241/) (2021)
- [Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control](https://mlsanity.com/paper/2103.06859/) (2021)
- [Dual Online Stein Variational Inference for Control and Dynamics](https://mlsanity.com/paper/2103.12890/) (2021)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [Learning Sampling Distributions for Model Predictive Control](https://mlsanity.com/paper/2212.02587/) (2022)
- [Efficient Dynamics Modeling in Interactive Environments with Koopman Theory](https://mlsanity.com/paper/2306.11941/) (2023)
- [Applications of the Free Energy Principle to Machine Learning and Neuroscience](https://mlsanity.com/paper/2107.00140/) (2021)
- [Mind the Model, Not the Agent: The Primacy Bias in Model-based RL](https://mlsanity.com/paper/2310.15017/) (2023)
- [Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization](https://mlsanity.com/paper/2105.12991/) (2021)
- [CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2112.07746/) (2021)
- [Constrained Stein Variational Trajectory Optimization](https://mlsanity.com/paper/2308.12110/) (2023)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Domain-Adversarial and Conditional State Space Model for Imitation Learning](https://mlsanity.com/paper/2001.11628/) (2020)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation](https://mlsanity.com/paper/2503.05238/) (2025)
- [A Simple Decentralized Cross-Entropy Method](https://mlsanity.com/paper/2212.08235/) (2022)
- [A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robots](https://mlsanity.com/paper/2103.08183/) (2021)
- [Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens](https://mlsanity.com/paper/2506.06261/) (2025)
- [Risk-sensitive control as inference with R'enyi divergence](https://mlsanity.com/paper/2411.01827/) (2024)
- [AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models](https://mlsanity.com/paper/2505.24784/) (2025)
- [Probabilistic Constrained Reinforcement Learning with Formal Interpretability](https://mlsanity.com/paper/2307.07084/) (2023)
- [A Contact Model based on Denoising Diffusion to Learn Variable Impedance Control for Contact-rich Manipulation](https://mlsanity.com/paper/2403.13221/) (2024)
- [Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Model](https://mlsanity.com/paper/2208.03936/) (2022)
- [Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency](https://mlsanity.com/paper/2412.12894/) (2024)

## Abstract (arXiv)

In recent studies on model-based reinforcement learning (MBRL), incorporating uncertainty in forward dynamics is a state-of-the-art strategy to enhance learning performance, making MBRLs competitive to cutting-edge model free methods, especially in simulated robotics tasks. Probabilistic ensembles with trajectory sampling (PETS) is a leading type of MBRL, which employs Bayesian inference to dynamics modeling and model predictive control (MPC) with stochastic optimization via the cross entropy method (CEM). In this paper, we propose a novel extension to the uncertainty-aware MBRL. Our main contributions are twofold: Firstly, we introduce a variational inference MPC, which reformulates various stochastic methods, including CEM, in a Bayesian fashion. Secondly, we propose a novel instance of the framework, called probabilistic action ensembles with trajectory sampling (PaETS). As a result, our Bayesian MBRL can involve multimodal uncertainties both in dynamics and optimal trajectories. In comparison to PETS, our method consistently improves asymptotic performance on several challenging locomotion tasks.