mlsanityData to 13 Oct 2026

Reinforcement learning with an expectile-based objective

arXiv 2602.09300 ↗First posted 10 Feb 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We consider the policy evaluation and control in a finite horizon reinforcement learning (RL) setting under an expectile-based objective. First, we derive the mean-squared error (MSE) and concentration bounds for the classic estimator of expectiles based on independent and identically distributed (i.i.d.) samples. To the best of our knowledge, expectiles have not been analyzed in the non-asymptotic regime and the bounds we derive may be of independent interest. Next, we analyze a Monte Carlo type estimator of expectile of the Markov chain underlying a given policy. We derive upper bounds that hold in expectation as well as with high probability for this estimator. Further, we show the order-optimality of our estimator by deriving a lower bound on expectile-based policy evaluation. For the problem of control, we adopt a policy gradient approach and derive a policy gradient theorem for expectiles. Using this result, we propose a gradient estimator with a Oleft(1/mright) mean-squared error bounds, where m is the number of trajectories. Further, under standard assumptions for policy gradient-type algorithms, we establish smoothness of the expectile-sensitive objective, in turn leading to stationary convergence rate bounds for the overall risk-sensitive policy gradient algorithm that we propose. Finally, we conduct numerical experiments to show the utility of expectiles on popular RL benchmarks.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 43 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

21 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Bootstrapping Expectiles in Reinforcement Learning6 Jun

Other references · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,310
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,257
  4. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 718
  5. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  6. 2013Risk-sensitive Reinforcement Learningcited by 129
  7. 2014Optimizing the CVaR via Samplingcited by 98
  8. 2015Policy Gradient for Coherent Risk Measurescited by 75
  9. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  10. 2015Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Controlcited by 34
  11. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  12. 2021Reinforcement Learning with Dynamic Convex Risk Measurescited by 20
8 more
  1. 2020Risk-Averse Learning by Temporal Difference Methodscited by 13
  2. 2023On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measurescited by 7
  3. 2021Policy Gradient Methods for Distortion Risk Measurescited by 6
  4. 2022A Risk-Sensitive Approach to Policy Optimizationcited by 6
  5. 2021Smoothed functional-based gradient algorithms for off-policy reinforcement learning: A non-asymptotic viewpointcited by 5
  6. 2022Ergodic Risk-sensitive control -- A surveycited by 4
  7. 2023Risk Estimation in a Markov Cost Process: Lower and Upper Boundscited by 2
  8. 2024Concentration Bounds for Optimized Certainty Equivalent Risk Estimationcited by 2

Built on by

Not cited yet.