mlsanityData to 7 Oct 2026

Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?

arXiv 2406.06309 ↗First posted 10 Jun 2024cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In deep Reinforcement Learning (RL), value functions are typically approximated using deep neural networks and trained via mean squared error regression objectives to fit the true value functions. Recent research has proposed an alternative approach, utilizing the cross-entropy classification objective, which has demonstrated improved performance and scalability of RL algorithms. However, existing study have not extensively benchmarked the effects of this replacement across various domains, as the primary objective was to demonstrate the efficacy of the concept across a broad spectrum of tasks, without delving into in-depth analysis. Our work seeks to empirically investigate the impact of such a replacement in an offline RL setup and analyze the effects of different aspects on performance. Through large-scale experiments conducted across a diverse range of tasks using different algorithms, we aim to gain deeper insights into the implications of this approach. Our results reveal that incorporating this change can lead to superior performance over state-of-the-art solutions for some algorithms in certain tasks, while maintaining comparable performance levels in other tasks, however for other algorithms this modification might lead to the dramatic performance drop. This findings are crucial for further application of classification approach in research and practical tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 45 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

38 of 45 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models8 Feb
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  3. 2023Anti-Exploration by Random Network Distillation31 Jan
  4. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  5. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  6. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  7. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  8. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  9. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL16 Mar
  10. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  11. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  12. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
5 more
  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2016Pixel Recurrent Neural Networkscited by 814
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  10. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 359
  11. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  12. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
9 more
  1. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  2. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  3. 2018LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Imagescited by 88
  4. 2018Improving Regression Performance with Distributional Lossescited by 66
  5. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  6. 2023Improving Deep Regression with Ordinal Entropycited by 44
  7. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  8. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flowscited by 21
  9. 2024Investigating the Histogram Loss in Regressioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Yes, Q-learning Helps Offline In-Context RLcited by 8
  2. 2024The Role of Deep Learning Regularizations on Actors in Offline RLcited by 2
  3. 2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 2
  4. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?
  2. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning
  3. 2025VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning
  4. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  5. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets
  6. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning
  7. 2021Offline Reinforcement Learning with Value-based Episodic Memory
  8. 2022Data Valuation for Offline Reinforcement Learning