Data to 7 Oct 2026

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

arXiv 1805.00909 ↗First posted 2 May 2018cs.LG cs.AI cs.ROcited by 466

Not on the map. Why?

Cited by 65 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.

Key facts. Cited by 466 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 8; 2020: 6; 2021: 70; 2022: 64; 2023: 56; 2024: 70; 2025: 86; 2026: 106.

466 papers.

Sort:

By area:

2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2019Causal Confusion in Imitation Learningcited by 286
2021GFlowNet Foundationscited by 286
2026Reinforcement Learning via Self-Distillationcited by 285
2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
2022Online Decision Transformercited by 194

38 more

2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Modelscited by 155
2023Protein Design with Guided Discrete Diffusioncited by 154
2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 125
2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
2026MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memorycited by 113
2022Language Model Cascadescited by 104
2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
2021A Survey on Interpretable Reinforcement Learningcited by 70
2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
2019Multi-Agent Adversarial Inverse Reinforcement Learningcited by 60
2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
2021Identifiability in inverse reinforcement learningcited by 55
2021Active Inference in Robotics and Artificial Agents: Survey and Challengescited by 52
2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
2024Variational Best-of-N Alignmentcited by 49
2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
2022Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?cited by 45
2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
2025On the Guidance of Flow Matchingcited by 44
2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42

50 most cited of 466 All 466 (JSON)