mlsanityData to 14 Oct 2026

Safe Reinforcement Learning with Linear Function Approximation

arXiv 2106.06239 ↗First posted 11 Jun 2021cs.LG stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

Safety in reinforcement learning has become increasingly important in recent years. Yet, existing solutions either fail to strictly avoid choosing unsafe actions, which may lead to catastrophic results in safety-critical systems, or fail to provide regret guarantees for settings where safety constraints need to be learned. In this paper, we address both problems by first modeling safety as an unknown linear cost function of states and actions, which must always fall below a certain threshold. We then present algorithms, termed SLUCB-QVI and RSLUCB-QVI, for episodic Markov decision processes (MDPs) with linear function approximation. We show that SLUCB-QVI and RSLUCB-QVI, while with no safety violation, achieve a tildeOleft(kappa√d3H3Tright) regret, nearly matching that of state-of-the-art unsafe algorithms, where H is the duration of each episode, d is the dimension of the feature mapping, kappa is a constant characterizing the safety constraints, and T is the total number of action plays. We further present numerical simulations that corroborate our theoretical findings.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

23 of 37 references.

References · 23

  1. 2016OpenAI Gymcited by 1,736
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2018Reward Constrained Policy Optimizationcited by 417
  4. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  5. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  6. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  7. 2020Projection-Based Constrained Policy Optimizationcited by 203
  8. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  9. 2020Exploration-Exploitation in Constrained MDPscited by 149
  10. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  11. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  12. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
11 more
  1. 2019Linear Stochastic Bandits Under Safety Constraintscited by 90
  2. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  3. 2019Provably Efficient Q-Learning with Low Switching Costcited by 78
  4. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 75
  5. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 74
  6. 2020Stochastic Bandits with Linear Constraintscited by 68
  7. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  8. 2020Constrained Upper Confidence Reinforcement Learningcited by 46
  9. 2020A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraintscited by 34
  10. 2020Conservative Exploration in Reinforcement Learningcited by 20
  11. 2020Decentralized Multi-Agent Linear Bandits with Safety Constraintscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 10; 2024: 7; 2025: 3; 2026: 8.

35 papers.

Sort:

By area:

  1. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 65
  2. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  3. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 31
  4. 2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 28
  5. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 27
  6. 2023A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraintscited by 17
  7. 2023Provably Safe Reinforcement Learning with Step-wise Violation Constraintscited by 11
  8. 2022Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RLcited by 10
  9. 2021A Reduction-Based Framework for Conservative Bandits and Reinforcement Learningcited by 9
  10. 2023Directional Optimism for Safe Linear Banditscited by 8
  11. 2022Provable Safe Reinforcement Learning with Binary Feedbackcited by 7
  12. 2023Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Explorationcited by 7
23 more
  1. 2025Provably Efficient RL for Linear MDPs under Instantaneous Safety Constraints in Non-Convex Feature Spacescited by 7
  2. 2024A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guaranteescited by 6
  3. 2025Provably Efficient RL under Episode-Wise Safety in Constrained MDPs with Linear Function Approximationcited by 6
  4. 2023Online Nonstochastic Control with Adversarial and Static Constraintscited by 5
  5. 2023Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraintscited by 5
  6. 2024Safety through feedback in Constrained RLcited by 5
  7. 2021Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehiclescited by 4
  8. 2023Provable Reset-free Reinforcement Learning by No-Regret Reductioncited by 3
  9. 2024Flipping-based Policy for Chance-Constrained Markov Decision Processescited by 3
  10. 2026Near-Optimal Primal-Dual Algorithm for Learning Linear Mixture CMDPs with Adversarial Rewardscited by 2
  11. 2026Learning Weakly Communicating Average-Reward CMDPs: Strong Duality and Improved Regretcited by 2
  12. 2023Anytime-Competitive Reinforcement Learning with Policy Priorcited by 1
  13. 2024Optimistic Safety for Online Convex Optimization with Unknown Linear Constraintscited by 1
  14. 2024Myopically Verifiable Probabilistic Certificates for Safe Control and Learningcited by 1
  15. 2025Constrained Online Decision-Making: A Unified Frameworkcited by 1
  16. 2026Near-Optimal Sample Complexity for Online Constrained MDPscited by 1
  17. 2023Semi-Infinitely Constrained Markov Decision Processes and Efficient Reinforcement Learningcited by 0
  18. 2024When to Localize? A Risk-Constrained Reinforcement Learning Approachcited by 0
  19. 2026Safe Reinforcement Learning with Preference-based Constraint Inferencecited by 0
  20. 2026Data-Driven Synthesis of Probabilistic Controlled Invariant Sets for Linear MDPscited by 0
  21. 2026A Queueing-Theoretic Framework for Dynamic Attack Surfaces: Data-Integrated Risk Analysis and Adaptive Defensecited by 0
  22. 2026Primal-Dual Policy Optimization for Linear CMDPs with Adversarial Lossescited by 0
  23. 2026Instance-Dependent Regret for CMDPs with Step-Wise Constraintscited by 0