mlsanityData to 7 Oct 2026

Outcome-Driven Reinforcement Learning via Variational Inference

arXiv 2104.10190 ↗First posted 20 Apr 2021cs.LG cs.AI stat.MEcited by 17

Not on the map. Why?

Abstract (arXiv)

While reinforcement learning algorithms provide automated acquisition of optimal policies, practical application of such methods requires a number of design decisions, such as manually designing reward functions that not only define the task, but also provide sufficient shaping to accomplish it. In this paper, we view reinforcement learning as inferring policies that achieve desired outcomes, rather than as a problem of maximizing rewards. To solve this inference problem, we establish a novel variational inference formulation that allows us to derive a well-shaped reward function which can be learned directly from environment interactions. From the corresponding variational objective, we also derive a new probabilistic Bellman backup operator and use it to develop an off-policy algorithm to solve goal-directed tasks. We empirically demonstrate that this method eliminates the need to hand-craft reward functions for a suite of diverse manipulation and locomotion tasks and leads to effective goal-directed behaviors.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 29 of its 64 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

29 of 64 references.

References · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  11. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  12. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
17 more
  1. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  2. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  3. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  4. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  5. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  6. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  7. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  8. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  9. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  10. 2019Information asymmetry in KL-regularized RLcited by 53
  11. 2016Unifying task specification in reinforcement learningcited by 41
  12. 2018VIREL: A Variational Inference Framework for Reinforcement Learningcited by 22
  13. 2019Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learningcited by 18
  14. 2014Optimal steering of a linear stochastic system to a final probability distribution, part IIcited by 17
  15. 2020Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learningcited by 12
  16. 2019Nonlinear Uncertainty Control with Iterative Covariance Steeringcited by 6
  17. 2019Nonlinear Covariance Control via Differential Dynamic Programmingcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 5; 2023: 4; 2024: 2; 2025: 2.

17 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  2. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  3. 2023Contrastive Difference Predictive Codingcited by 35
  4. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  5. 2021Contrastive Active Inferencecited by 21
  6. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  7. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  8. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  9. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 18
  10. 2021Understanding the World Through Actioncited by 17
  11. 2021Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learningcited by 15
  12. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
5 more
  1. 2024On Divergence Measures for Training GFlowNetscited by 6
  2. 2022Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspectivecited by 0
  3. 2023Learning Generative Models with Goal-conditioned Reinforcement Learningcited by 0
  4. 2025A representational framework for learning and encoding structurally enriched trajectories in complex agent environmentscited by 0
  5. 2025Visual Pre-Training on Unlabeled Images using Reinforcement Learningcited by 0