mlsanityData to 7 Oct 2026

High-Dimensional Continuous Control Using Generalized Advantage Estimation

arXiv 1506.02438 ↗First posted 8 Jun 2015cs.LG cs.RO cs.SYcited by 1,916

Not on the map. Why?

Cited by 206 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Policy gradient methods are an appealing approach in reinforcement learning because they directly optimize the cumulative reward and can straightforwardly be used with nonlinear function approximators such as neural networks. The two main challenges are the large number of samples typically required, and the difficulty of obtaining stable and steady improvement despite the nonstationarity of the incoming data. We address the first challenge by using value functions to substantially reduce the variance of policy gradient estimates at the cost of some bias, with an exponentially-weighted estimator of the advantage function that is analogous to TD(lambda). We address the second challenge by using trust region optimization procedure for both the policy and the value function, which are represented by neural networks. Our approach yields strong empirical results on highly challenging 3D locomotion tasks, learning running gaits for bipedal and quadrupedal simulated robots, and learning a policy for getting the biped to stand up from starting out lying on the ground. In contrast to a body of prior work that uses hand-crafted policy representations, our neural network policies map directly from raw kinematics to joint torques. Our algorithm is fully model-free, and the amount of simulated experience required for the learning tasks on 3D bipeds corresponds to 1-2 weeks of real time.

Key facts. Cited by 1,916 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 4; 2019: 5; 2020: 10; 2021: 218; 2022: 211; 2023: 255; 2024: 251; 2025: 479; 2026: 483.

1,916 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  8. 2023Mastering Diverse Domains through World Modelscited by 1,193
  9. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
38 more
  1. 2025Process Reinforcement through Implicit Rewardscited by 478
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  5. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  6. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  8. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  10. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  11. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  12. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  13. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  14. 2026Reinforcement Learning via Self-Distillationcited by 285
  15. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  16. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  17. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  18. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  19. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  20. 2024Diffusion Policy Policy Optimizationcited by 255
  21. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  22. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  23. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  24. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  25. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  26. 2021Mastering Atari Games with Limited Datacited by 235
  27. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  28. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  29. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  30. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  31. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  32. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  33. 2022Online Decision Transformercited by 194
  34. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  35. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  36. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  37. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  38. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175

50 most cited of 1,916 All 1,916 (JSON)