mlsanityData to 14 Oct 2026

Learning to Score Behaviors for Guided Policy Optimization

arXiv 1906.04349 ↗First posted 11 Jun 2019cs.LG stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space. We show that by utilizing the dual formulation of the WD, we can learn score functions over policy behaviors that can in turn be used to lead policy optimization towards (or away from) (un)desired behaviors. Combined with smoothed WDs, the dual formulation allows us to devise efficient algorithms that take stochastic gradient descent steps through WD regularizers. We incorporate these regularizers into two novel on-policy algorithms, Behavior-Guided Policy Gradient and Behavior-Guided Evolution Strategies, which we demonstrate can outperform existing methods in a variety of challenging environments. We also provide an open source demo.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 8; 2022: 7; 2023: 8; 2024: 5; 2025: 2; 2026: 3.

34 papers.

Sort:

By area:

  1. 2020Primal Wasserstein Imitation Learningcited by 121
  2. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  3. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  4. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  5. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  6. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 25
  7. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  8. 2021Wasserstein Unsupervised Reinforcement Learningcited by 22
  9. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 20
  10. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 19
  11. 2024Group-Aware Coordination Graph for Multi-Agent Reinforcement Learningcited by 18
  12. 2025Wasserstein Policy Optimizationcited by 13
22 more
  1. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  2. 2021Towards an Understanding of Default Policies in Multitask Policy Optimizationcited by 8
  3. 2022Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actionscited by 8
  4. 2023Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policiescited by 8
  5. 2024Inferring Latent Temporal Sparse Coordination Graph for Multi-Agent Reinforcement Learningcited by 7
  6. 2023Provably Convergent Policy Optimization via Metric-aware Trust Region Methodscited by 6
  7. 2024A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Pointscited by 5
  8. 2024Optimal Transport-Assisted Risk-Sensitive Q-Learningcited by 4
  9. 2026Wasserstein Proximal Policy Gradientcited by 4
  10. 2022Truly Deterministic Policy Optimizationcited by 3
  11. 2022Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processescited by 3
  12. 2023Iteratively Learn Diverse Strategies with State Distance Informationcited by 3
  13. 2023A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learningcited by 3
  14. 2025AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimizationcited by 3
  15. 2021Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transportcited by 1
  16. 2021Agent Spacescited by 1
  17. 2022Minimum Description Length Controlcited by 0
  18. 2022Transfer RL via the Undo Maps Formalismcited by 0
  19. 2023Wasserstein Diversity-Enriched Regularizer for Hierarchical Reinforcement Learningcited by 0
  20. 2024Phasic Diversity Optimization for Population-Based Reinforcement Learningcited by 0
  21. 2026Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimizationcited by 0
  22. 2026Implicit Neural Representations of Individual Behaviorcited by 0