# Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems

arXiv: https://arxiv.org/abs/2404.15199 · HTML: https://mlsanity.com/paper/2404.15199/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [Constrained Variational Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2201.11927/) (2022)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Reachability Constrained Reinforcement Learning](https://mlsanity.com/paper/2205.07536/) (2022)
- [Neurosymbolic Reinforcement Learning with Formally Verified Exploration](https://mlsanity.com/paper/2009.12612/) (2020)
- [Control Regularization for Reduced Variance Reinforcement Learning](https://mlsanity.com/paper/1905.05380/) (2019)
- [Towards Safe Reinforcement Learning with a Safety Editor Policy](https://mlsanity.com/paper/2201.12427/) (2022)
- [Deep Reinforcement Learning for Closed-Loop Blood Glucose Control](https://mlsanity.com/paper/2009.09051/) (2020)
- [Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes](https://mlsanity.com/paper/2210.10691/) (2022)
- [You Only Live Once: Single-Life Reinforcement Learning](https://mlsanity.com/paper/2210.08863/) (2022)
- [Learning Transferable Domain Priors for Safe Exploration in Reinforcement Learning](https://mlsanity.com/paper/1909.04307/) (2019)
- [Driving Reinforcement Learning with Models](https://mlsanity.com/paper/1911.04400/) (2019)
- [Reinforcement Learning Based on Real-Time Iteration NMPC](https://mlsanity.com/paper/2005.05225/) (2020)

## Built on by (most cited first, 6 of 6)

- [Can RL Improve Generalization of LLM Agents? An Empirical Study](https://mlsanity.com/paper/2603.12011/) (2026)
- [A Novel Framework for Uncertainty-Driven Adaptive Exploration](https://mlsanity.com/paper/2509.03219/) (2025)
- [Mission-Aligned Learning-Informed Control of Autonomous Systems: Formulation and Foundations](https://mlsanity.com/paper/2507.04356/) (2025)
- [Constraint-Aware Reinforcement Learning via Adaptive Action Scaling](https://mlsanity.com/paper/2510.11491/) (2025)
- [Safe Online Learning via Smooth Safety-Structured Policy Composition](https://mlsanity.com/paper/2606.31320/) (2026)
- [HetGPS: Scalable Graph Multi-Agent Reinforcement Learning with Physics-Anchored Adaptive Safety for EV Charging](https://mlsanity.com/paper/2608.00679/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) is a powerful method for controlling dynamic systems, but its learning mechanism can lead to unpredictable actions that undermine the safety of critical systems. Here, we propose RL with Adaptive Regularization (RL-AR), an algorithm that enables safe RL exploration by combining the RL policy with a policy regularizer that hard-codes the safety constraints. RL-AR performs policy combination via a "focus module," which determines the appropriate combination depending on the state--relying more on the safe policy regularizer for less-exploited states while allowing unbiased convergence for well-exploited states. In a series of critical control applications, we demonstrate that RL-AR not only ensures safety during training but also achieves a return competitive with the standards of model-free RL that disregards safety.