# Kernel-Based Safe Exploration in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2605.22207 · HTML: https://mlsanity.com/paper/2605.22207/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 21 May 2026.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 28 of 28)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [Kernel methods through the roof: handling billions of points efficiently](https://mlsanity.com/paper/2006.10350/) (2020)
- [A Measure-Theoretic Approach to Kernel Conditional Mean Embeddings](https://mlsanity.com/paper/2002.03689/) (2020)
- [OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research](https://mlsanity.com/paper/2305.09304/) (2023)
- [Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments](https://mlsanity.com/paper/2209.15090/) (2022)
- [Constrained Update Projection Approach to Safe Policy Optimization](https://mlsanity.com/paper/2209.07089/) (2022)
- [Beyond variance reduction: Understanding the true impact of baselines on policy optimization](https://mlsanity.com/paper/2008.13773/) (2020)
- [A Rigorous Theory of Conditional Mean Embeddings](https://mlsanity.com/paper/1912.00671/) (2019)
- [Kernel-Based Reinforcement Learning: A Finite-Time Analysis](https://mlsanity.com/paper/2004.05599/) (2020)
- [Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions](https://mlsanity.com/paper/1903.09885/) (2019)
- [Practical Kernel-Based Reinforcement Learning](https://mlsanity.com/paper/1407.5358/) (2014)
- [Data-driven verification and synthesis of stochastic systems via barrier certificates](https://mlsanity.com/paper/2111.10330/) (2021)
- [Data-Driven Distributionally Robust Safety Verification Using Barrier Certificates and Conditional Mean Embeddings](https://mlsanity.com/paper/2403.10497/) (2024)
- [Maximum Mean Discrepancy Distributionally Robust Nonlinear Chance-Constrained Optimization with Finite-Sample Guarantee](https://mlsanity.com/paper/2204.11564/) (2022)
- [Distributionally Robust Optimal and Safe Control of Stochastic Systems via Kernel Conditional Mean Embedding](https://mlsanity.com/paper/2304.00644/) (2023)
- [Sparse Approximation of a Kernel Mean](https://mlsanity.com/paper/1503.00323/) (2015)
- [Data-Efficient Control Barrier Function Refinement](https://mlsanity.com/paper/2303.05973/) (2023)
- [Stochastic Optimal Control via Hilbert Space Embeddings of Distributions](https://mlsanity.com/paper/2103.12759/) (2021)
- [Learning-Based Shielding for Safe Autonomy under Unknown Dynamics](https://mlsanity.com/paper/2410.07359/) (2024)
- [Data-Driven Stochastic Optimal Control Using Kernel Gradients](https://mlsanity.com/paper/2209.09205/) (2022)

## Abstract (arXiv)

Safety has been a major concern when deploying deep reinforcement learning algorithms in the real world. A promising direction that ensures that the learned policy does not visit unsafe regions is to learn a barrier function along with the policy. A barrier is a function from states to reals that assigns low values to the initial states, high values to the unsafe states, and decreases in expectation on each transition; such a function can be used to bound the probability of reaching unsafe states. Previous attempts learned a barrier function directly from exploration data, but this required either large amounts of data or restrictions on the system dynamics. In this paper, we show how kernel embeddings can be used to learn barrier functions during deep reinforcement learning for stochastic systems with unknown dynamics. Our algorithm, kernel-based safe exploration (KBSE), learns an optimal policy and a barrier simultaneously during exploration. The barriers are computed iteratively, represented as conditional mean embeddings, and provide better probabilistic safety guarantees with more exploration. The exploration algorithm uses the learned barrier functions to identify safety violations. In the case of violation, it intervenes to modify the unsafe action to a safe action, thereby ensuring that the exploration is restricted to actions that bound the probability of reaching unsafe states. We evaluate KBSE on several complex continuous control benchmarks. Experimental results establish our new algorithm to be suitable for synthesizing control policies that are probabilistically safe without degradation in reward accumulation.