# Toward Integrating Adaptive Experience Replay and Online Uncertainty Estimation in Safe Actor-Critic Optimal Control

arXiv: https://arxiv.org/abs/2608.04732 · HTML: https://mlsanity.com/paper/2608.04732/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 4 of its 39 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 4 of 4)

- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [Optimal control barrier functions for RL based safe powertrain control](https://mlsanity.com/paper/2405.11391/) (2024)
- [Self-Organizing Dual-Buffer Adaptive Clustering Experience Replay (SODACER) for Safe Reinforcement Learning in Optimal Control](https://mlsanity.com/paper/2601.06540/) (2026)

## Abstract (arXiv)

Safe actor-critic control often treats barrier filtering, uncertainty estimation, and experience replay as separate modules, even though each changes the data used for learning and control. We develop an integrated architecture in which the uncertainty estimate updates the obstacle geometry used by a control barrier function, filter interventions and estimation residuals determine replay priority, and the critic learns from the executed rather than nominal action. We instantiate the architecture on a two-dimensional robot-navigation task with corrupted obstacle measurements and compare six component-matched configurations under common training budgets, random seeds, sensor streams, exploration, and disturbances. Evaluation includes a moderate post-training test, an eleven-level perception-noise sweep, and an exploratory extreme-stress test at multiplier 6.0. In the extreme test, the integrated configuration recorded no contacts and reached the goal in all five evaluation seeds. Its mean cost was 7.63±0.44 and its obstacle-belief root-mean-square error was 3.52±0.55 cm. The uncertainty-estimation ablation also recorded no contacts but reached the goal in four of five seeds, with mean cost 8.96±2.08 and belief error 11.08±1.23 cm. A finite-training bound clarifies replay exposure, and a robust barrier condition states the required estimation-error and feasibility assumptions. The results support coupling estimation, safety filtering, and replay on this benchmark; broader safety and convergence claims require further study.