Planning for Change: Reinforcement Learning Combined with Bounded Extremum Seeking for Robotic Control under Distribution Shift
Not on the map. Why?
Abstract (arXiv)
Reinforcement learning has shown strong performance in robotic manipulation, but learned policies often degrade in performance when test conditions differ from the training distribution. This limitation is especially important for reliable robot planning and control in contact-rich tasks such as pushing and pick-and-place, where changes in goals, contact conditions, or robot dynamics can drive the system out-of-distribution at inference time. In this paper, we investigate a hybrid controller that combines reinforcement learning with bounded extremum seeking (ES) to improve robustness under such conditions. In the proposed approach, deep deterministic policy gradient (DDPG) policies are trained under standard conditions on the robotic pushing and pick-and-place tasks, and are then combined with bounded ES during deployment. The RL policy provides fast manipulation behavior, while bounded ES ensures robustness of the overall controller to time variations when operating conditions depart from those seen during training. The resulting controller is evaluated under several out-of-distribution settings, including time-varying goals and spatially varying friction patches. Crucially, under reasonable assumptions, we prove finite time RL-based convergence of a robotic arm to an object after which bounded ES achieves finite-time convergence to a bounded velocity moving target. We provide analytic formulas for both convergence times, and an ability to tune the convergence times by our choice of gains.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 36 references that are in the map. First posted on arXiv 1 Apr 2026.
Builds on
18 of 36 references.
References · 18
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
- 2018Quantifying Generalization in Reinforcement Learningcited by 345
- 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
- 2018Residual Reinforcement Learning for Robot Controlcited by 242
- 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
- 2022Partially Observable Markov Decision Processes in Robotics: A Surveycited by 81
- 2021CARL: A Benchmark for Contextual and Adaptive Reinforcement Learningcited by 21
- 2021Dynamic Grasping with Reachability and Motion Awarenesscited by 15
- 2025LocoFormer: Generalist Locomotion via Long-context Adaptationcited by 14
- 2023Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flightcited by 13
6 more
- 2023Deep Reinforcement Learning for Robotic Pushing and Picking in Cluttered Environmentcited by 10
- 2023Logic programming for deliberative robotic task planningcited by 8
- 2024Model-Based Transfer Learning for Contextual Reinforcement Learningcited by 7
- 2021Online Policies for Real-Time Control Using MRAC-RLcited by 5
- 2025Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seekingcited by 3
- 2017A Fast Stochastic Contact Model for Planar Pushing and Grasping: Theory and Experimental Validationcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.