# Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations

arXiv: https://arxiv.org/abs/2609.10866 · HTML: https://mlsanity.com/paper/2609.10866/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 9 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 15 of 15)

- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising](https://mlsanity.com/paper/1805.06530/) (2018)
- [Delving into adversarial attacks on deep policies](https://mlsanity.com/paper/1705.06452/) (2017)
- [Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks](https://mlsanity.com/paper/1701.04143/) (2017)
- [Certified Adversarial Robustness for Deep Reinforcement Learning](https://mlsanity.com/paper/1910.12908/) (2019)
- [Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning](https://mlsanity.com/paper/2210.05927/) (2022)
- [CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing](https://mlsanity.com/paper/2106.09292/) (2021)
- [Policy Smoothing for Provably Robust Reinforcement Learning](https://mlsanity.com/paper/2106.11420/) (2021)
- [Whatever Does Not Kill Deep Reinforcement Learning, Makes It Stronger](https://mlsanity.com/paper/1712.09344/) (2017)
- [Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/2004.06496/) (2020)
- [Reward Certification for Policy Smoothed Reinforcement Learning](https://mlsanity.com/paper/2312.06436/) (2023)

## Abstract (arXiv)

Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under lp-norm-bounded state adversarial perturbations (1≤ p <∞). By introducing a φ-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter β independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.