mlsanityData to 14 Oct 2026

Safe-RULE: Safe Reinforcement UnLEarning

arXiv 2606.09559 ↗First posted 8 Jun 2026cs.LG cs.AI cs.CRcited by 0

On the map: LLM safety and security → Machine unlearning

also in 1 more
Abstract (arXiv)

Offline safe reinforcement learning (Safe RL) enables policy learning without online interactions, making it suitable for safety-critical systems such as robotics systems. However, its reliance on static datasets exposes offline Safe RL to data poisoning attacks, where adversaries inject malicious samples that compromise safety and induce unsafe policy behavior. In this work, we propose a new learning paradigm, named safe reinforcement unlearning (Safe-RULE), used as a defense framework to remove the influence of poisoned data without retraining from scratch or requiring access to the original training environment. We further extend reinforcement unlearning to offline Safe RL by explicitly accounting for both task performance and safety constraints during the unlearning process. Experiments across benchmark Safe RL tasks demonstrate that our approach effectively enhances safety performance against data poisoning attacks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 33 of its 38 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

33 of 38 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Verifying Machine Unlearning with Explainable AI20 Nov
  2. 2024Applying sparse autoencoders to unlearn knowledge in language models25 Oct
  3. 2024Do Unlearning Methods Remove Information from Language Model Weights?11 Oct
  4. 2024Silver Linings in the Shadows: Harnessing Membership Inference for Machine Unlearning1 Jul
  5. 2024Second-Order Information Matters: Revisiting Machine Unlearning for Large Language Models13 Mar
  6. 2024Unlearning Traces the Influential Training Data of Language Models26 Jan
  7. 2023Reinforcement Unlearning26 Dec
  8. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2022Editing Models with Task Arithmeticcited by 1,120
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  6. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  7. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  8. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  9. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  10. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  11. 2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
  12. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
13 more
  1. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  2. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
  3. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 79
  4. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  5. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  6. 2019Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signalscited by 50
  7. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  8. 2021Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learningcited by 29
  9. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 29
  10. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasetscited by 25
  11. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstrationcited by 14
  12. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  13. 2025PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Safe Reinforcement Learning with Minimal Supervision
  2. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  3. 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy
  4. 2025Skill-based Safe Reinforcement Learning with Risk Planning
  5. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  6. 2026Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning
  7. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  8. 2026Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning