Data to 7 Oct 2026

Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation

arXiv 1708.05144 ↗First posted 17 Aug 2017cs.LGcited by 119

Not on the map. Why?

Abstract (arXiv)

In this work, we propose to apply trust region optimization to deep reinforcement learning using a recently proposed Kronecker-factored approximation to the curvature. We extend the framework of natural policy gradient and propose to optimize both the actor and the critic using Kronecker-factored approximate curvature (K-FAC) with trust region; hence we call our method Actor Critic using Kronecker-Factored Trust Region (ACKTR). To the best of our knowledge, this is the first scalable trust region natural gradient method for actor-critic methods. It is also a method that learns non-trivial tasks in continuous control as well as discrete control policies directly from raw pixel inputs. We tested our approach across discrete domains in Atari games as well as continuous domains in the MuJoCo environment. With the proposed methods, we are able to achieve higher rewards and a 2- to 3-fold improvement in sample efficiency on average, compared to previous state-of-the-art on-policy actor-critic methods. Code is available at https://github.com/openai/baselines

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 3; 2021: 34; 2022: 26; 2023: 21; 2024: 9; 2025: 15; 2026: 9.

119 papers.

Sort:

By area:

2025Learning to Reason under Off-Policy Guidancecited by 266
2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
2018Multi-Agent Generative Adversarial Imitation Learningcited by 99
2019Multi-Agent Adversarial Inverse Reinforcement Learningcited by 60
2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 54
2021Adversarially Guided Actor-Criticcited by 47
2021Deep Multiagent Reinforcement Learning: Challenges and Directionscited by 42
2021Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Surveycited by 34
2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29

38 more

2022Combining Modular Skills in Multitask Learningcited by 27
2020Robust Imitation Learning from Noisy Demonstrationscited by 26
2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 22
2021Multi-agent Reinforcement Learning for Cooperative Lane Changing of Connected and Autonomous Vehicles in Mixed Trafficcited by 20
2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 19
2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
2021Safe Reinforcement Learning with Chance-constrained Model Predictive Controlcited by 18
2021Learning Practically Feasible Policies for Online 3D Bin Packingcited by 16
2022Modeling Human Exploration Through Resource-Rational Reinforcement Learningcited by 13
2024Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learningcited by 13
2022A Survey on Reinforcement Learning Methods in Character Animationcited by 12
2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
2022Learning Physically Realizable Skills for Online Packing of General 3D Shapescited by 12
2022General Policy Evaluation and Improvement by Learning to Identify Few But Crucial Statescited by 10
2022Reinforcement Learning Algorithms: An Overview and Classificationcited by 10
2023Correcting discount-factor mismatch in on-policy policy gradient methodscited by 9
2023Efficient Symbolic Policy Learning with Differentiable Symbolic Expressioncited by 9
2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
2022Online 3D Bin Packing Reinforcement Learning Solution with Buffercited by 8
2023Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatmentcited by 8
2025Learning Boltzmann Generators via Constrained Mass Transportcited by 8
2023Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithmscited by 7
2020Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Criticcited by 6
2021Kronecker-factored Quasi-Newton Methods for Deep Learningcited by 6
2021Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertaintycited by 6
2022Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Reviewcited by 6
2023Provably Convergent Policy Optimization via Metric-aware Trust Region Methodscited by 6
2024Fast Deep Hedging with Second-Order Optimizationcited by 6
2021Dealing with Non-Stationarity in MARL via Trust-Region Decompositioncited by 5
2021Episodic Policy Gradient Trainingcited by 5
2022Mask Atari for Deep Reinforcement Learning as POMDP Benchmarkscited by 5
2023Assessment of Reinforcement Learning Algorithms for Nuclear Power Plant Fuel Optimizationcited by 5
2023AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learningcited by 5
2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
2025A Cascading Cooperative Multi-agent Framework for On-ramp Merging Control Integrating Large Language Modelscited by 5
2022Learning to Constrain Policy Optimization with Virtual Trust Regioncited by 4

50 most cited of 119 All 119 (JSON)