mlsanityData to 7 Oct 2026

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

arXiv 1910.00177 ↗First posted 1 Oct 2019cs.LG stat.MLcited by 750

Not on the map. Why?

Cited by 352 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.

Key facts. Cited by 750 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 12; 2021: 43; 2022: 74; 2023: 104; 2024: 137; 2025: 163; 2026: 216.

750 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  10. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  12. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
38 more
  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  4. 2026Reinforcement Learning via Self-Distillationcited by 285
  5. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  7. 2025Improving Video Generation with Human Feedbackcited by 267
  8. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  9. 2024Diffusion Policy Policy Optimizationcited by 255
  10. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  11. 2023Pretraining Language Models with Human Preferencescited by 245
  12. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  13. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  14. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 218
  15. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  16. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  17. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  18. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  19. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  20. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  21. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  22. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  23. 2025Agentic Reinforced Policy Optimizationcited by 166
  24. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  25. 2025Flow Q-Learningcited by 162
  26. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  27. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  28. 2021Offline RL Without Off-Policy Evaluationcited by 150
  29. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  30. 2023Structured World Models from Human Videoscited by 144
  31. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  32. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  33. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  34. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  35. 2023Controlled Decoding from Language Modelscited by 133
  36. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  37. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  38. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121

50 most cited of 750 All 750 (JSON)