mlsanityData to 14 Oct 2026

Learning safe, constrained policies via imitation learning: Connection to Probabilistic Inference and a Naive Algorithm

arXiv 2507.06780 ↗First posted 9 Jul 2025cs.LG cs.MAcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

also in 1 more
Abstract (arXiv)

This article introduces an imitation learning method for learning maximum entropy policies that comply with constraints demonstrated by expert trajectories executing a task. The formulation of the method takes advantage of results connecting performance to bounds for the KL-divergence between demonstrated and learned policies, and its objective is rigorously justified through a connection to a probabilistic inference framework for reinforcement learning, incorporating the reinforcement learning objective and the objective to abide by constraints in an entropy maximization setting. The proposed algorithm optimizes the learning objective with dual gradient descent, supporting effective and stable training. Experiments show that the proposed method can learn effective policy models for constraints-abiding behaviour, in settings with multiple constraints of different types, accommodating different modalities of demonstrated behaviour, and with abilities to generalize.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 23 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

16 of 23 references.

References · 16

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2021Convex optimizationcited by 2,515
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2017Constrained Policy Optimizationcited by 983
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  8. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  9. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  10. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  11. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 111
  12. 2017Robust Imitation of Diverse Behaviorscited by 86
4 more
  1. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  2. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 70
  3. 2020Inverse Constrained Reinforcement Learningcited by 46
  4. 2020Accelerating Safe Reinforcement Learning with Constraint-mismatched Policiescited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  3. 2022Weighted Maximum Entropy Inverse Reinforcement Learning
  4. 2018Fast Policy Learning through Imitation and Reinforcement
  5. 2021Policy Gradient Bayesian Robust Optimization for Imitation Learning
  6. 2019Reward-Conditioned Policies
  7. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  8. 2022Imitation Learning by State-Only Distribution Matching