# Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting

arXiv: https://arxiv.org/abs/2602.04131 · HTML: https://mlsanity.com/paper/2602.04131/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 23 of 23)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Time Limits in Reinforcement Learning](https://mlsanity.com/paper/1712.00378/) (2017)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [Discount Factor as a Regularizer in Reinforcement Learning](https://mlsanity.com/paper/2007.02040/) (2020)
- [Unifying task specification in reinforcement learning](https://mlsanity.com/paper/1609.01995/) (2016)
- [Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach](https://mlsanity.com/paper/1902.02893/) (2019)
- [How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies](https://mlsanity.com/paper/1512.02011/) (2015)
- [Discounted Reinforcement Learning Is Not an Optimization Problem](https://mlsanity.com/paper/1910.02140/) (2019)
- [Distributional Reinforcement Learning for Multi-Dimensional Reward Functions](https://mlsanity.com/paper/2110.13578/) (2021)
- [Reinforcement Learning with Dynamic Convex Risk Measures](https://mlsanity.com/paper/2112.13414/) (2021)
- [Foundations of Multivariate Distributional Reinforcement Learning](https://mlsanity.com/paper/2409.00328/) (2024)
- [Distributionally Robust Markov Decision Processes and their Connection to Risk Measures](https://mlsanity.com/paper/2007.13103/) (2020)
- [Optimizing Return Distributions with Distributional Dynamic Programming](https://mlsanity.com/paper/2501.13028/) (2025)
- [G-Learner and GIRL: Goal Based Wealth Management with Reinforcement Learning](https://mlsanity.com/paper/2002.10990/) (2020)
- [Reinforcement Learning with Non-Exponential Discounting](https://mlsanity.com/paper/2209.13413/) (2022)
- [Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning](https://mlsanity.com/paper/2501.02087/) (2025)
- [Practical Risk Measures in Reinforcement Learning](https://mlsanity.com/paper/1908.08379/) (2019)
- [Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning](https://mlsanity.com/paper/2507.03900/) (2025)
- [Relaxed Equilibria for Time-Inconsistent Markov Decision Processes](https://mlsanity.com/paper/2307.04227/) (2023)

## Built on by (most cited first, 1 of 1)

- [Temporal Preference Concepts and their Functions in a Large Language Model](https://mlsanity.com/paper/2606.05194/) (2026)

## Abstract (arXiv)

Distributional reinforcement learning (RL) is a powerful framework increasingly adopted in safety-critical domains for its ability to optimize risk-sensitive objectives. However, the role of the discount factor is often overlooked, as it is typically treated as a fixed parameter of the Markov decision process or tunable hyperparameter, with little consideration of its effect on the learned policy. In the literature, it is well-known that the discounting function plays a major role in characterizing time preferences of an agent, which an exponential discount factor cannot fully capture. Building on this insight, we propose a novel framework that supports flexible discounting of future rewards and optimization of risk measures in distributional RL. We provide a technical analysis of the optimality of our algorithms, show that our multi-horizon extension fixes issues raised with existing methodologies, and validate the robustness of our methods through extensive experiments. Our results highlight that discounting is a cornerstone in decision-making problems for capturing more expressive temporal and risk preferences profiles, with potential implications for real-world safety-critical applications.