# Robust Temporal Difference Learning for Critical Domains

arXiv: https://arxiv.org/abs/1901.08021 · HTML: https://mlsanity.com/paper/1901.08021/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2019.

## Built on by (most cited first, 7 of 7)

- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation](https://mlsanity.com/paper/2305.18388/) (2023)
- [Efficient Action Robust Reinforcement Learning with Probabilistic Policy Execution Uncertainty](https://mlsanity.com/paper/2307.07666/) (2023)
- [Bounded Robustness in Reinforcement Learning via Lexicographic Objectives](https://mlsanity.com/paper/2209.15320/) (2022)
- [SQT -- std Q-target](https://mlsanity.com/paper/2402.05950/) (2024)
- [Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation](https://mlsanity.com/paper/2506.16753/) (2025)
- [Tackling Snow-Induced Challenges: Safe Autonomous Lane-Keeping with Robust Reinforcement Learning](https://mlsanity.com/paper/2512.12987/) (2025)

## Abstract (arXiv)

We present a new Q-function operator for temporal difference (TD) learning methods that explicitly encodes robustness against significant rare events (SRE) in critical domains. The operator, which we call the kappa-operator, allows to learn a robust policy in a model-based fashion without actually observing the SRE. We introduce single- and multi-agent robust TD methods using the operator kappa. We prove convergence of the operator to the optimal robust Q-function with respect to the model using the theory of Generalized Markov Decision Processes. In addition we prove convergence to the optimal Q-function of the original MDP given that the probability of SREs vanishes. Empirical evaluations demonstrate the superior performance of kappa-based TD methods both in the early learning phase as well as in the final converged stage. In addition we show robustness of the proposed method to small model errors, as well as its applicability in a multi-agent context.