Constrained Policy Gradient Method for Safe and Fast Reinforcement Learning: a Neural Tangent Kernel Based Approach
Not on the map. Why?
Abstract (arXiv)
This paper presents a constrained policy gradient algorithm. We introduce constraints for safe learning with the following steps. First, learning is slowed down (lazy learning) so that the episodic policy change can be computed with the help of the policy gradient theorem and the neural tangent kernel. Then, this enables us the evaluation of the policy at arbitrary states too. In the same spirit, learning can be guided, ensuring safety via augmenting episode batches with states where the desired action probabilities are prescribed. Finally, exogenous discounted sum of future rewards (returns) can be computed at these specific state-action pairs such that the policy network satisfies constraints. Computing the returns is based on solving a system of linear equations (equality constraints) or a constrained quadratic program (inequality constraints, regional constraints). Simulation results suggest that adding constraints (external information) to the learning can improve learning in terms of speed and transparency reasonably if constraints are appropriately selected. The efficiency of the constrained learning was demonstrated with a shallow and wide ReLU network in the Cartpole and Lunar Lander OpenAI gym environments. The main novelty of the paper is giving a practical use of the neural tangent kernel in reinforcement learning.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 19 Jul 2021.
Builds on
22 of 45 references.
References · 22
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2021Convex optimizationcited by 2,479
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2016OpenAI Gymcited by 1,684
- 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
- 2017A Brief Survey of Deep Reinforcement Learningcited by 466
- 2017Robust Adversarial Reinforcement Learningcited by 439
- 2018Reward Constrained Policy Optimizationcited by 397
- 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
- 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
- 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
- 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 109
10 more
- 2018Stagewise Safe Bayesian Optimization with Gaussian Processescited by 79
- 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
- 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
- 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
- 2018Expected Policy Gradients for Reinforcement Learningcited by 28
- 2018Exploration versus exploitation in reinforcement learning: a stochastic control approachcited by 17
- 2019Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methodscited by 11
- 2020Balancing a CartPole System with Reinforcement Learning -- A Tutorialcited by 7
- 2020Reinforcement Learning via Gaussian Processes with Neural Network Dual Kernelscited by 3
- 2020Solving The Lunar Lander Problem under Uncertainty using Reinforcement Learningcited by 3
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 1.
2 papers.