mlsanityData to 14 Oct 2026

Progressive Safeguards for Safe and Model-Agnostic Reinforcement Learning

arXiv 2410.24096 ↗First posted 31 Oct 2024cs.LG cs.LOcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

In this paper we propose a formal, model-agnostic meta-learning framework for safe reinforcement learning. Our framework is inspired by how parents safeguard their children across a progression of increasingly riskier tasks, imparting a sense of safety that is carried over from task to task. We model this as a meta-learning process where each task is synchronized with a safeguard that monitors safety and provides a reward signal to the agent. The safeguard is implemented as a finite-state machine based on a safety specification; the reward signal is formally shaped around this specification. The safety specification and its corresponding safeguard can be arbitrarily complex and non-Markovian, which adds flexibility to the training process and explainability to the learned policy. The design of the safeguard is manual but it is high-level and model-agnostic, which gives rise to an end-to-end safe learning approach with wide applicability, from pixel-level game control to language model fine-tuning. Starting from a given set of safety specifications (tasks), we train a model such that it can adapt to new specifications using only a small number of training samples. This is made possible by our method for efficiently transferring safety bias between tasks, which effectively minimizes the number of safety violations. We evaluate our framework in a Minecraft-inspired Gridworld, a VizDoom game environment, and an LLM fine-tuning application. Agents trained with our approach achieve near-minimal safety violations, while baselines are shown to underperform.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 43 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

25 of 43 references.

References · 25

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2019Batch Policy Learning under Constraintscited by 244
  5. 2017Neural Episodic Controlcited by 151
  6. 2020Conservative Safety Critics for Explorationcited by 113
  7. 2016Model-Free Episodic Controlcited by 102
  8. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  9. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  10. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  11. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 75
  12. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 69
13 more
  1. 2019Reinforcement Learning with Convex Constraintscited by 60
  2. 2018Logically-Constrained Reinforcement Learningcited by 51
  3. 2018ViZDoom Competitions: Playing Doom from Pixelscited by 51
  4. 2020Neurosymbolic Reinforcement Learning with Formally Verified Explorationcited by 49
  5. 2020Cautious Reinforcement Learning with Logical Constraintscited by 45
  6. 2014Safe Exploration of State and Action Spaces in Reinforcement Learningcited by 38
  7. 2019Certified Reinforcement Learning with Logic Guidancecited by 36
  8. 2020Verifiably Safe Exploration for End-to-End Reinforcement Learningcited by 34
  9. 2019Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functionscited by 21
  10. 2013Scaling Up Robust MDPs by Reinforcement Learningcited by 17
  11. 2018Logically-Constrained Neural Fitted Q-Iterationcited by 17
  12. 2019DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learningcited by 9
  13. 2023Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specificationscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe Meta-Reinforcement Learning via Information Space Reachability
  2. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  3. 2024Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning
  4. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  5. 2026Constrained Meta Reinforcement Learning with Provable Test-Time Safety
  6. 2019Parenting: Safe Reinforcement Learning from Human Input
  7. 2025Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?
  8. 2025Probabilistic Shielding for Safe Reinforcement Learning