mlsanityData to 7 Oct 2026

A CMDP-within-online framework for Meta-Safe Reinforcement Learning

arXiv 2405.16601 ↗First posted 26 May 2024cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Meta-reinforcement learning has widely been used as a learning-to-learn framework to solve unseen tasks with limited experience. However, the aspect of constraint violations has not been adequately addressed in the existing works, making their application restricted in real-world settings. In this paper, we study the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the first provable guarantees in this important setting. We obtain task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the task-averaged optimality gap and constraint satisfaction improve with task-similarity in a static environment or task-relatedness in a dynamic environment. Several technical challenges arise when making this framework practical. To this end, we propose a meta-algorithm that performs inexact online learning on the upper bounds of within-task optimality gap and constraint violations estimated by off-policy stationary distribution corrections. Furthermore, we enable the learning rates to be adapted for every task and extend our approach to settings with a competing dynamically changing oracle. Finally, experiments are conducted to demonstrate the effectiveness of our approach.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 69 of its 106 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

69 of 106 references.

References · 69

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2019Introduction to Online Convex Optimizationcited by 793
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  7. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  9. 2019A Theory of Regularized Markov Decision Processescited by 279
  10. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  11. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  12. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
38 more
  1. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  2. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  3. 2019Batch Policy Learning under Constraintscited by 242
  4. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  5. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  6. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  7. 2015The Benefit of Multitask Representation Learningcited by 194
  8. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  9. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  10. 2013Non-stationary Stochastic Optimizationcited by 173
  11. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  12. 2020Exploration-Exploitation in Constrained MDPscited by 146
  13. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  14. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  15. 2019Online Meta-Learningcited by 137
  16. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  17. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  18. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  19. 2018Stochastic subgradient method converges on tame functionscited by 120
  20. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  21. 2015Online Optimization : Competing with Dynamic Comparatorscited by 99
  22. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  23. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  24. 2020Dynamic Regret of Convex and Smooth Functionscited by 90
  25. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 78
  26. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 75
  27. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 69
  28. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 69
  29. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  30. 2016Tracking Slowly Moving Clairvoyant: Optimal Dynamic Regret of Online Learning with True and Noisy Gradientcited by 65
  31. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  32. 2016Error bounds, quadratic growth, and linear convergence of proximal methodscited by 59
  33. 2019ES-MAML: Simple Hessian-Free Meta Learningcited by 57
  34. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 55
  35. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  36. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  37. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  38. 2016Improved Dynamic Regret for Non-degenerate Functionscited by 46

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 8.

12 papers.

  1. 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparatorcited by 2
  2. 2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
  3. 2025Safe In-Context Reinforcement Learningcited by 2
  4. 2026Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Artcited by 1
  5. 2026Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbedcited by 1
  6. 2026Constrained Meta Reinforcement Learning with Provable Test-Time Safetycited by 1
  7. 2025TLXML: Task-Level Explanation of Meta-Learning via Influence Functionscited by 0
  8. 2026Toward Adaptive Grid Resilience: A Gradient-Free Meta-RL Framework for Critical Load Restorationcited by 0
  9. 2026From Cumulative Constraints to Adaptive Runtime Safety Control for Nonstationary Reinforcement Learningcited by 0
  10. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 0
  11. 2026Proactive Context-Forecasted Safety Constraints for Nonstationary Reinforcement Learningcited by 0
  12. 2026Safe Meta-Reinforcement Learning via Information Space Reachabilitycited by 0