mlsanityData to 7 Oct 2026

Stop Regressing: Training Value Functions via Classification for Scalable Deep RL

arXiv 2403.03950 ↗First posted 6 Mar 2024cs.LG cs.AI stat.MLcited by 136

Not on the map. Why?

Cited by 25 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Value functions are a central component of deep reinforcement learning (RL). These functions, parameterized by neural networks, are trained using a mean squared error regression objective to match bootstrapped target values. However, scaling value-based RL methods that use regression to large networks, such as high-capacity Transformers, has proven challenging. This difficulty is in stark contrast to supervised learning: by leveraging a cross-entropy classification loss, supervised methods have scaled reliably to massive networks. Observing this discrepancy, in this paper, we investigate whether the scalability of deep RL can also be improved simply by using classification in place of regression for training value functions. We demonstrate that value functions trained with categorical cross-entropy significantly improves performance and scalability in a variety of domains. These include: single-task RL on Atari 2600 games with SoftMoEs, multi-task RL on Atari with large-scale ResNets, robotic manipulation with Q-transformers, playing Chess without search, and a language-agent Wordle task with high-capacity Transformers, achieving state-of-the-art results on these domains. Through careful analysis, we show that the benefits of categorical cross-entropy primarily stem from its ability to mitigate issues inherent to value-based RL, such as noisy targets and non-stationarity. Overall, we argue that a simple shift to training value functions with categorical cross-entropy can yield substantial improvements in the scalability of deep RL at little-to-no cost.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Builds on 57 of its 78 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

57 of 78 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2023Mastering Diverse Domains through World Modelscited by 1,193
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
38 more
  1. 2016Pixel Recurrent Neural Networkscited by 814
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  5. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 359
  6. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  7. 2023From Sparse to Soft Mixtures of Expertscited by 245
  8. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  9. 2022Multi-Game Decision Transformerscited by 212
  10. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  11. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  12. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  13. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  14. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  15. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  16. 2018Generalization and Regularization in DQNcited by 117
  17. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  18. 2018LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Imagescited by 88
  19. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  20. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  21. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  22. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  23. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68
  24. 2018Improving Regression Performance with Distributional Lossescited by 66
  25. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  26. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  27. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  28. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  29. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  30. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  31. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 45
  32. 2023Improving Deep Regression with Ordinal Entropycited by 44
  33. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  34. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  35. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  36. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  37. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  38. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 56; 2026: 44.

136 papers.

Sort:

By area:

  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  3. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  4. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  5. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  6. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  7. 2025Horizon Reduction Makes RL Scalablecited by 55
  8. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  9. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  10. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  11. 2025TAPNext: Tracking Any Point (TAP) as Next Token Predictioncited by 35
  12. 2024Hidden Biases of End-to-End Driving Datasetscited by 30
38 more
  1. 2025Self-Improving Embodied Foundation Modelscited by 28
  2. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  3. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  4. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  5. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  6. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  7. 2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 17
  8. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  9. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  10. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  11. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actionscited by 15
  12. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  13. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  14. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  15. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretrainingcited by 13
  16. 2025Improving Transformer World Models for Data-Efficient RLcited by 13
  17. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  18. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 12
  19. 2025Value Flowscited by 12
  20. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulationcited by 11
  21. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  22. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  23. 2024Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Modelscited by 10
  24. 2024Fourier Head: Helping Large Language Models Learn Complex Probability Distributionscited by 10
  25. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuningcited by 10
  26. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  27. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrastscited by 9
  28. 2025Discrete Codebook World Models for Continuous Controlcited by 9
  29. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  30. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  31. 2024Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learningcited by 8
  32. 2024The Central Role of the Loss Function in Reinforcement Learningcited by 8
  33. 2025Yes, Q-learning Helps Offline In-Context RLcited by 8
  34. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  35. 2025Simulus: Combining Improvements in Sample-Efficient World Model Agentscited by 6
  36. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  37. 2025Open-ended Hierarchical Streaming Video Understanding with Vision Language Modelscited by 6
  38. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6

50 most cited of 136 All 136 (JSON)