Optimal Transport-Assisted Risk-Sensitive Q-Learning
Not on the map. Why?
Abstract (arXiv)
The primary goal of reinforcement learning is to develop decision-making policies that prioritize optimal performance without considering risk or safety. In contrast, safe reinforcement learning aims to mitigate or avoid unsafe states. This paper presents a risk-sensitive Q-learning algorithm that leverages optimal transport theory to enhance the agent safety. By integrating optimal transport into the Q-learning framework, our approach seeks to optimize the policy's expected return while minimizing the Wasserstein distance between the policy's stationary distribution and a predefined risk distribution, which encapsulates safety preferences from domain experts. We validate the proposed algorithm in a Gridworld environment. The results indicate that our method significantly reduces the frequency of visits to risky states and achieves faster convergence to a stable policy compared to the traditional Q-learning algorithm.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 17 Jun 2024.
Builds on
28 of 38 references.
References · 28
- 2017Constrained Policy Optimizationcited by 928
- 2017Safe Reinforcement Learning via Shieldingcited by 427
- 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
- 2018Safe Exploration in Continuous Action Spacescited by 237
- 2020Projection-Based Constrained Policy Optimizationcited by 195
- 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
- 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
- 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
- 2018Learning-based Model Predictive Control for Safe Explorationcited by 116
- 2019Safe Reinforcement Learning Using Robust MPCcited by 72
- 2019Finite Sample Analysis of Stochastic System Identificationcited by 67
16 more
- 2020Cautious Reinforcement Learning with Logical Constraintscited by 44
- 2015Shield Synthesis: Runtime Enforcement for Reactive Systemscited by 41
- 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
- 2018Policy Optimization as Wasserstein Gradient Flowscited by 33
- 2019Deep Reinforcement Learning with Enhanced Safety for Autonomous Highway Drivingcited by 21
- 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulationcited by 21
- 2022A Survey on Reinforcement Learning in Aviation Applicationscited by 19
- 2018Optimal Transport on Discrete Domainscited by 16
- 2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 16
- 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
- 2023Risk-Aware Reinforcement Learning through Optimal Transport Theorycited by 7
- 2023Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learningcited by 6
- 2024The Synergy Between Optimal Transport Theory and Multi-Agent Reinforcement Learningcited by 5
- 2022Safe Exploration Method for Reinforcement Learning under Existence of Disturbancecited by 3
- 2023Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environmentscited by 3
- 2024How does Your RL Agent Explore? An Optimal Transport Analysis of Occupancy Measure Trajectoriescited by 1
Built on by
Citing papers per year: 2025: 3.
3 papers.