# VIREL: A Variational Inference Framework for Reinforcement Learning

arXiv: https://arxiv.org/abs/1811.01132 · HTML: https://mlsanity.com/paper/1811.01132/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2018.

## Built on by (most cited first, 22 of 22)

- [Constrained Variational Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2201.11927/) (2022)
- [A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning](https://mlsanity.com/paper/2102.13176/) (2021)
- [Online Variational Filtering and Parameter Learning](https://mlsanity.com/paper/2110.13549/) (2021)
- [Outcome-Driven Reinforcement Learning via Variational Inference](https://mlsanity.com/paper/2104.10190/) (2021)
- [Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.17258/) (2021)
- [Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming](https://mlsanity.com/paper/2201.08484/) (2022)
- [Probabilistic Inference in Reinforcement Learning Done Right](https://mlsanity.com/paper/2311.13294/) (2023)
- [Batch Bayesian optimisation via density-ratio estimation with guarantees](https://mlsanity.com/paper/2209.10715/) (2022)
- [Variational Delayed Policy Optimization](https://mlsanity.com/paper/2405.14226/) (2024)
- [Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions](https://mlsanity.com/paper/2411.00465/) (2024)
- [Habitizing Diffusion Planning for Efficient and Effective Decision Making](https://mlsanity.com/paper/2502.06401/) (2025)
- [HyQE: Ranking Contexts with Hypothetical Query Embeddings](https://mlsanity.com/paper/2410.15262/) (2024)
- [Hierarchical Variational Imitation Learning of Control Programs](https://mlsanity.com/paper/1912.12612/) (2019)
- [Factorised Active Inference for Strategic Multi-Agent Interactions](https://mlsanity.com/paper/2411.07362/) (2024)
- [Posterior Sampling with Delayed Feedback for Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2310.18919/) (2023)
- [Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning](https://mlsanity.com/paper/2510.08526/) (2025)
- [Tensorizing flows: a tool for variational inference](https://mlsanity.com/paper/2305.02460/) (2023)
- [Reinforcement Learning for Topic Models](https://mlsanity.com/paper/2305.04843/) (2023)
- [Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning](https://mlsanity.com/paper/2506.01639/) (2025)
- [Concurrent Credit Assignment for Data-efficient Reinforcement Learning](https://mlsanity.com/paper/2205.12020/) (2022)
- [Minimum Description Length Control](https://mlsanity.com/paper/2207.08258/) (2022)
- [Boltzmann Reinforcement Learning for Noise resilience in Analog Ising Machines](https://mlsanity.com/paper/2602.09162/) (2026)

## Abstract (arXiv)

Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in pseudo-likelihood methods and difficulties learning deterministic policies in maximum entropy RL based approaches. We propose VIREL, a novel, theoretically grounded probabilistic inference framework for RL that utilises a parametrised action-value function to summarise future dynamics of the underlying MDP. This gives VIREL a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference and the ability to optimise value functions and policies in separate, iterative steps. In applying variational expectation-maximisation to VIREL we thus show that the actor-critic algorithm can be reduced to expectation-maximisation, with policy improvement equivalent to an E-step and policy evaluation to an M-step. We then derive a family of actor-critic methods from VIREL, including a scheme for adaptive exploration. Finally, we demonstrate that actor-critic algorithms from this family outperform state-of-the-art methods based on soft value functions in several domains.