mlsanityData to 7 Oct 2026

Q-Value Weighted Regression: Reinforcement Learning with Limited Data

arXiv 2102.06782 ↗First posted 12 Feb 2021cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Sample efficiency and performance in the offline setting have emerged as significant challenges of deep reinforcement learning. We introduce Q-Value Weighted Regression (QWR), a simple RL algorithm that excels in these aspects. QWR is an extension of Advantage Weighted Regression (AWR), an off-policy actor-critic algorithm that performs very well on continuous control tasks, also in the offline setting, but has low sample efficiency and struggles with high-dimensional observation spaces. We perform an analysis of AWR that explains its shortcomings and use these insights to motivate QWR. We show experimentally that QWR matches the state-of-the-art algorithms both on tasks with continuous and discrete actions. In particular, QWR yields results on par with SAC on the MuJoCo suite and - with the same set of hyperparameters - yields results on par with a highly tuned Rainbow implementation on a set of Atari games. We also verify that QWR performs well in the offline RL setting.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

31 of 48 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
19 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  3. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2019Model-Based Reinforcement Learning for Ataricited by 495
  8. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  9. 2018Maximum a Posteriori Policy Optimisationcited by 318
  10. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  11. 2020Critic Regularized Regressioncited by 269
  12. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  13. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  14. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  15. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  16. 2019When to use parametric models in reinforcement learning?cited by 118
  17. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
  18. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
  19. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 0; 2024: 1; 2025: 1.

4 papers.

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2022On the Global Convergence of Fitted Q-Iteration with Two-layer Neural Network Parametrizationcited by 3
  3. 2025COLSON: Controllable Learning-Based Social Navigation via Diffusion-Based Reinforcement Learningcited by 2
  4. 2024Shared learning of powertrain control policies for vehicle fleetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization
  2. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  3. 2022State Advantage Weighting for Offline RL
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  5. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  6. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  7. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
  8. 2021Offline RL Without Off-Policy Evaluation