# A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning

arXiv: https://arxiv.org/abs/2401.02325 · HTML: https://mlsanity.com/paper/2401.02325/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 4 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 17 of 17)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [An Alternative Probabilistic Interpretation of the Huber Loss](https://mlsanity.com/paper/1911.02088/) (2019)
- [Generalized Huber Loss for Robust Learning and its Efficient Minimization for a Robust Statistics](https://mlsanity.com/paper/2108.12627/) (2021)
- [GMAC: A Distributional Perspective on Actor-Critic Framework](https://mlsanity.com/paper/2105.11366/) (2021)
- [Gamma and Vega Hedging Using Deep Distributional Reinforcement Learning](https://mlsanity.com/paper/2205.05614/) (2022)
- [Robust Reinforcement Learning with Distributional Risk-averse formulation](https://mlsanity.com/paper/2206.06841/) (2022)
- [Robust Losses for Learning Value Functions](https://mlsanity.com/paper/2205.08464/) (2022)
- [On the 1-Wasserstein Distance between Location-Scale Distributions and the Effect of Differential Privacy](https://mlsanity.com/paper/2304.14869/) (2023)
- [POPO: Pessimistic Offline Policy Optimization](https://mlsanity.com/paper/2012.13682/) (2020)
- [Deep Huber quantile regression networks](https://mlsanity.com/paper/2306.10306/) (2023)
- [Robust Estimation and Shrinkage in Ultrahigh Dimensional Expectile Regression with Heavy Tails and Variance Heterogeneity](https://mlsanity.com/paper/1909.09302/) (2019)
- [Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.01768/) (2023)
- [Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning](https://mlsanity.com/paper/2310.10818/) (2023)

## Built on by (most cited first, 2 of 2)

- [EX-DRL: Hedging Against Heavy Losses with EXtreme Distributional Reinforcement Learning](https://mlsanity.com/paper/2408.12446/) (2024)
- [Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping](https://mlsanity.com/paper/2610.13909/) (2026)

## Abstract (arXiv)

Distributional Reinforcement Learning (RL) estimates return distribution mainly by learning quantile values via minimizing the quantile Huber loss function, entailing a threshold parameter often selected heuristically or via hyperparameter search, which may not generalize well and can be suboptimal. This paper introduces a generalized quantile Huber loss function derived from Wasserstein distance (WD) calculation between Gaussian distributions, capturing noise in predicted (current) and target (Bellman-updated) quantile values. Compared to the classical quantile Huber loss, this innovative loss function enhances robustness against outliers. Notably, the classical Huber loss function can be seen as an approximation of our proposed loss, enabling parameter adjustment by approximating the amount of noise in the data during the learning process. Empirical tests on Atari games, a common application in distributional RL, and a recent hedging strategy using distributional RL, validate the effectiveness of our proposed loss function and its potential for parameter adjustments in distributional RL. The implementation of the proposed loss function is available here.