mlsanityData to 13 Oct 2026

Sample Efficient Actor-Critic with Experience Replay

arXiv 1611.01224 ↗First posted 3 Nov 2016cs.LGcited by 186

Not on the map. Why?

Cited by 43 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents an actor-critic deep reinforcement learning agent with experience replay that is stable, sample efficient, and performs remarkably well on challenging environments, including the discrete 57-game Atari domain and several continuous control problems. To achieve this, the paper introduces several innovations, including truncated importance sampling with bias correction, stochastic dueling network architectures, and a new trust region policy optimization method.

Key facts. Cited by 186 papers in the map (data through 13 Oct 2026). First posted on arXiv 3 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 3; 2021: 56; 2022: 38; 2023: 25; 2024: 24; 2025: 19; 2026: 19.

186 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
  2. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 145
  3. 2018Self-Imitation Learningcited by 135
  4. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  5. 2025Flow Matching Policy Gradientscited by 90
  6. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  7. 2025ExGRPO: Learning to Reason from Experiencecited by 52
  8. 2021Adversarially Guided Actor-Criticcited by 47
  9. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 42
  10. 2021Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithmcited by 39
  11. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  12. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 39
38 more
  1. 2018Sample-Efficient Imitation Learning via Generative Adversarial Netscited by 36
  2. 2021Reinforcement Learning for Ridesharing: An Extended Surveycited by 35
  3. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 33
  4. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
  5. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  6. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
  7. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  8. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  9. 2021Unbiased Asymmetric Reinforcement Learning under Partial Observabilitycited by 25
  10. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  11. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
  12. 2021CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Modelscited by 20
  13. 2022Combining Evolution and Deep Reinforcement Learning for Policy Search: a Surveycited by 20
  14. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  15. 2021Safe Reinforcement Learning with Chance-constrained Model Predictive Controlcited by 18
  16. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  17. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  18. 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
  19. 2024SAPG: Split and Aggregate Policy Gradientscited by 17
  20. 2021Learning Reward Machines: A Study in Partially Observable Reinforcement Learningcited by 16
  21. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  22. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  23. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
  24. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  25. 2021Direct Advantage Estimationcited by 11
  26. 2022A Meta-Reinforcement Learning Algorithm for Causal Discoverycited by 11
  27. 2022Automated Play-Testing Through RL Based Human-Like Play-Styles Generationcited by 11
  28. 2023VA-learning as a more efficient alternative to Q-learningcited by 11
  29. 2023Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Riskcited by 11
  30. 2022General Policy Evaluation and Improvement by Learning to Identify Few But Crucial Statescited by 10
  31. 2022Reinforcement Learning Algorithms: An Overview and Classificationcited by 10
  32. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  33. 2021Model-free Policy Learning with Reward Gradientscited by 9
  34. 2021PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replaycited by 9
  35. 2022Automated Reinforcement Learning: An Overviewcited by 9
  36. 2022Analysis of Stochastic Processes through Replay Bufferscited by 9
  37. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  38. 2021Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experiencecited by 8

50 most cited of 186 All 186 (JSON)