mlsanityData to 14 Oct 2026

Rectifying Regression in Reinforcement Learning

arXiv 2510.00885 ↗First posted 1 Oct 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper investigates the impact of the loss function in value-based methods for reinforcement learning through an analysis of underlying prediction objectives. We theoretically show that mean absolute error is a better prediction objective than the traditional mean squared error for controlling the learned policy's suboptimality gap. Furthermore, we present results that different loss functions are better aligned with these different regression objectives: binary and categorical cross-entropy losses with the mean absolute error and squared loss with the mean squared error. We then provide empirical evidence that algorithms minimizing these cross-entropy losses can outperform those based on the squared loss in linear reinforcement learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 31 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

15 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning11 Feb

Other references · 14

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  4. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  5. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  6. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  7. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  8. 2018Improving Regression Performance with Distributional Lossescited by 67
  9. 2021Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcited by 53
  10. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  11. 2017Generalized Self-Concordant Functions: A Recipe for Newton-Type Methodscited by 33
  12. 2016Fast Rates for General Unbounded Loss Functions: from ERM to Generalized Bayescited by 27
2 more
  1. 2024Investigating the Histogram Loss in Regressioncited by 11
  2. 2024Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Eluder dimension: localise it!cited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024The Central Role of the Loss Function in Reinforcement Learning
  2. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning
  3. 2025Calibrated Value-Aware Model Learning with Probabilistic Environment Models
  4. 2024A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learning
  5. 2024On the Limited Representational Power of Value Functions and its Links to Statistical (In)Efficiency
  6. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
  7. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms
  8. 2025Balancing Interpretability and Performance in Reinforcement Learning: An Adaptive Spectral Based Linear Approach