# Planning for Change: Reinforcement Learning Combined with Bounded Extremum Seeking for Robotic Control under Distribution Shift

arXiv: https://arxiv.org/abs/2604.01142 · HTML: https://mlsanity.com/paper/2604.01142/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 18 of 18)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Partially Observable Markov Decision Processes in Robotics: A Survey](https://mlsanity.com/paper/2209.10342/) (2022)
- [CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning](https://mlsanity.com/paper/2110.02102/) (2021)
- [LocoFormer: Generalist Locomotion via Long-context Adaptation](https://mlsanity.com/paper/2509.23745/) (2025)
- [Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight](https://mlsanity.com/paper/2306.09852/) (2023)
- [Dynamic Grasping with Reachability and Motion Awareness](https://mlsanity.com/paper/2103.10562/) (2021)
- [Deep Reinforcement Learning for Robotic Pushing and Picking in Cluttered Environment](https://mlsanity.com/paper/2302.10717/) (2023)
- [Logic programming for deliberative robotic task planning](https://mlsanity.com/paper/2301.07550/) (2023)
- [Online Policies for Real-Time Control Using MRAC-RL](https://mlsanity.com/paper/2103.16551/) (2021)
- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seeking](https://mlsanity.com/paper/2510.02490/) (2025)
- [A Fast Stochastic Contact Model for Planar Pushing and Grasping: Theory and Experimental Validation](https://mlsanity.com/paper/1705.10664/) (2017)

## Built on by (most cited first, 2 of 2)

- [Mahalanobis-Guided Latent OOD Detection for Hybrid ES-DRL Control in Time-Varying Systems](https://mlsanity.com/paper/2606.11474/) (2026)
- [Bidirectional Autoregressive Latent Diffusion for Forward and Inverse Magnetohydrodynamics](https://mlsanity.com/paper/2606.29620/) (2026)

## Abstract (arXiv)

Reinforcement learning has shown strong performance in robotic manipulation, but learned policies often degrade in performance when test conditions differ from the training distribution. This limitation is especially important for reliable robot planning and control in contact-rich tasks such as pushing and pick-and-place, where changes in goals, contact conditions, or robot dynamics can drive the system out-of-distribution at inference time. In this paper, we investigate a hybrid controller that combines reinforcement learning with bounded extremum seeking (ES) to improve robustness under such conditions. In the proposed approach, deep deterministic policy gradient (DDPG) policies are trained under standard conditions on the robotic pushing and pick-and-place tasks, and are then combined with bounded ES during deployment. The RL policy provides fast manipulation behavior, while bounded ES ensures robustness of the overall controller to time variations when operating conditions depart from those seen during training. The resulting controller is evaluated under several out-of-distribution settings, including time-varying goals and spatially varying friction patches. Crucially, under reasonable assumptions, we prove finite time RL-based convergence of a robotic arm to an object after which bounded ES achieves finite-time convergence to a bounded velocity moving target. We provide analytic formulas for both convergence times, and an ability to tune the convergence times by our choice of gains.