mlsanityData to 14 Oct 2026

Optimal Transport-Guided Safety in Temporal Difference Reinforcement Learning

arXiv 2502.16328 ↗First posted 22 Feb 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

The primary goal of reinforcement learning is to develop decision-making policies that prioritize optimal performance, frequently without considering safety. In contrast, safe reinforcement learning seeks to reduce or avoid unsafe behavior. This paper views safety as taking actions with more predictable consequences under environment stochasticity and introduces a temporal difference algorithm that uses optimal transport theory to quantify the uncertainty associated with actions. By integrating this uncertainty score into the decision-making objective, the agent is encouraged to favor actions with more predictable outcomes. We theoretically prove that our algorithm leads to a reduction in the probability of visiting unsafe states. We evaluate the proposed algorithm on several case studies in the presence of various forms of environment uncertainty. The results demonstrate that our method not only provides safer behavior but also maintains the performance. A Python implementation of our algorithm is available at hrefhttps://github.com/SAILRIT/Risk-averse-TD-Learninghttps://github.com/SAILRIT/OT-guided-TD-Learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 22 of its 37 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

22 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Optimal Transport-Assisted Risk-Sensitive Q-Learning17 Jun

Other references · 21

  1. 2017Constrained Policy Optimizationcited by 983
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  4. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  5. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  6. 2018Learning to Drive in a Daycited by 273
  7. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  8. 2021Recent Advances in Reinforcement Learning in Financecited by 110
  9. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  10. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
  11. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  12. 2019Open-Sourced Reinforcement Learning Environments for Surgical Roboticscited by 26
9 more
  1. 2021Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learningcited by 15
  2. 2018Exploration by Distributional Reinforcement Learningcited by 13
  3. 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
  4. 2023Risk-Aware Reinforcement Learning through Optimal Transport Theorycited by 8
  5. 2023Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learningcited by 7
  6. 2025Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learningcited by 4
  7. 2022Safe Exploration Method for Reinforcement Learning under Existence of Disturbancecited by 3
  8. 2021Risk-Averse Decision Making Under Uncertaintycited by 2
  9. 2024A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertaintycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Optimal Transport Meets Reinforcement Learning: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  2. 2025Probabilistic Shielding for Safe Reinforcement Learning
  3. 2026Evaluation Metrics for Safe Reinforcement Learning
  4. 2025Safety Representations for Safer Policy Learning
  5. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  6. 2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time
  7. 2024Revisiting Safe Exploration in Safe Reinforcement learning
  8. 2025Safe Planning and Policy Optimization via World Model Learning