mlsanityData to 7 Oct 2026

The Central Role of the Loss Function in Reinforcement Learning

arXiv 2409.12799 ↗First posted 19 Sep 2024stat.ML cs.LG math.STcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper illustrates the central role of loss functions in data-driven decision making, providing a comprehensive survey on their influence in cost-sensitive classification (CSC) and reinforcement learning (RL). We demonstrate how different regression loss functions affect the sample efficiency and adaptivity of value-based decision making algorithms. Across multiple settings, we prove that algorithms using the binary cross-entropy loss achieve first-order bounds scaling with the optimal policy's cost and are much more efficient than the commonly used squared loss. Moreover, we prove that distributional algorithms using the maximum likelihood loss achieve second-order bounds scaling with the policy variance and are even sharper than first-order bounds. This in particular proves the benefits of distributional RL. We hope that this paper serves as a guide analyzing decision making algorithms with varying loss functions, and can inspire the reader to seek out better loss functions to improve any decision making algorithm.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 49 of its 77 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

49 of 77 references.

References · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  8. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  9. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  10. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  11. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  12. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
37 more
  1. 2024Nemotron-4 340B Technical Reportcited by 143
  2. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  3. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  5. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  6. 2018Practical Contextual Bandits with Regression Oraclescited by 92
  7. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  8. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  9. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  10. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  11. 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
  12. 2020Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functionscited by 71
  13. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  14. 2018Improving Regression Performance with Distributional Lossescited by 66
  15. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  16. 2021Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcited by 53
  17. 2020Fast Rates for Contextual Linear Optimizationcited by 43
  18. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  19. 2020Logistic Q-Learningcited by 34
  20. 2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
  21. 2018Learning to Optimize Join Queries With Deep Reinforcement Learningcited by 32
  22. 2022On the Complexity of Adversarial Decision Makingcited by 32
  23. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  24. 2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 29
  25. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  26. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  27. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  28. 2017Small-loss bounds for online learning with partial informationcited by 18
  29. 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
  30. 2023On the Convergence and Sample Complexity Analysis of Deep Q-Networks with ε-Greedy Explorationcited by 17
  31. 2023Efficient Model-Free Exploration in Low-Rank MDPscited by 16
  32. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
  33. 2024The Power of Resets in Online Reinforcement Learningcited by 14
  34. 2021Provably Correct Optimization and Exploration with Non-linear Policiescited by 13
  35. 2024Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Boundscited by 11
  36. 2024Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learningcited by 8
  37. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 1.

8 papers.

  1. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  2. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  3. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  4. 2024Second Order Bounds for Contextual Bandits with Function Approximationcited by 9
  5. 2025Catoni Contextual Bandits are Robust to Heavy-tailed Rewardscited by 3
  6. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
  7. 2025Efficient Controllable Diffusion via Optimal Classifier Guidancecited by 2
  8. 2026Eluder dimension: localise it!cited by 2