# Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seeking

arXiv: https://arxiv.org/abs/2510.02490 · HTML: https://mlsanity.com/paper/2510.02490/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Partially Observable Markov Decision Processes in Robotics: A Survey](https://mlsanity.com/paper/2209.10342/) (2022)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning](https://mlsanity.com/paper/2110.02102/) (2021)
- [Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight](https://mlsanity.com/paper/2306.09852/) (2023)
- [Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL](https://mlsanity.com/paper/2012.09737/) (2020)
- [Online Policies for Real-Time Control Using MRAC-RL](https://mlsanity.com/paper/2103.16551/) (2021)
- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [Force Push: Robust Single-Point Pushing with Force Feedback](https://mlsanity.com/paper/2401.17517/) (2024)
- [Predictive control for nonlinear stochastic systems: Closed-loop guarantees with unbounded noise](https://mlsanity.com/paper/2407.13257/) (2024)
- [Recursive Averaging with Application to Bio-Inspired 3D Source Seeking](https://mlsanity.com/paper/2203.11928/) (2022)

## Built on by (most cited first, 4 of 4)

- [Planning for Change: Reinforcement Learning Combined with Bounded Extremum Seeking for Robotic Control under Distribution Shift](https://mlsanity.com/paper/2604.01142/) (2026)
- [Mahalanobis-Guided Latent OOD Detection for Hybrid ES-DRL Control in Time-Varying Systems](https://mlsanity.com/paper/2606.11474/) (2026)
- [Bidirectional Autoregressive Latent Diffusion for Forward and Inverse Magnetohydrodynamics](https://mlsanity.com/paper/2606.29620/) (2026)
- [Physics Foundation Models for Efficient Control of Fluid Flows](https://mlsanity.com/paper/2610.15173/) (2026)

## Abstract (arXiv)

In this paper, we study the use of robust model independent bounded extremum seeking (ES) feedback control to improve the robustness of deep reinforcement learning (DRL) controllers for a class of nonlinear time-varying systems. DRL has the potential to learn from large datasets to quickly control or optimize the outputs of many-parameter systems, but its performance degrades catastrophically when the system model changes rapidly over time. Bounded ES can handle time-varying systems with unknown control directions, but its convergence speed slows down as the number of tuned parameters increases and, like all local adaptive methods, it can get stuck in local minima. We demonstrate that together, DRL and bounded ES result in a hybrid controller whose performance exceeds the sum of its parts with DRL taking advantage of historical data to learn how to quickly control a many-parameter system to a desired setpoint while bounded ES ensures its robustness to time variations. We present a numerical study of a general time-varying system and a combined ES-DRL controller for automatic tuning of the Low Energy Beam Transport section at the Los Alamos Neutron Science Center linear particle accelerator.