# Performance-Driven Controller Tuning via Derivative-Free Reinforcement Learning

arXiv: https://arxiv.org/abs/2209.04854 · HTML: https://mlsanity.com/paper/2209.04854/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 11 Sep 2022.

## Builds on (most cited first, 18 of 18)

- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Objective Mismatch in Model-based Reinforcement Learning](https://mlsanity.com/paper/2002.04523/) (2020)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Smooth Exploration for Robotic Reinforcement Learning](https://mlsanity.com/paper/2005.05719/) (2020)
- [Goal-Aware Prediction: Learning to Model What Matters](https://mlsanity.com/paper/2007.07170/) (2020)
- [Learning Convex Optimization Control Policies](https://mlsanity.com/paper/1912.09529/) (2019)
- [Automatic LQR Tuning Based on Gaussian Process Global Optimization](https://mlsanity.com/paper/1605.01950/) (2016)
- [Deep Neuroevolution of Recurrent and Discrete World Models](https://mlsanity.com/paper/1906.08857/) (2019)
- [Performance-oriented model learning for data-driven MPC design](https://mlsanity.com/paper/1904.10839/) (2019)
- [From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion](https://mlsanity.com/paper/2011.11722/) (2020)
- [Goal-Driven Dynamics Learning via Bayesian Optimization](https://mlsanity.com/paper/1703.09260/) (2017)
- [From self-tuning regulators to reinforcement learning and back again](https://mlsanity.com/paper/1906.11392/) (2019)
- [Data-efficient Auto-tuning with Bayesian Optimization: An Industrial Control Study](https://mlsanity.com/paper/1812.06325/) (2018)
- [Learning to Predict Without Looking Ahead: World Models Without Forward Prediction](https://mlsanity.com/paper/1910.13038/) (2019)
- [Numerically Stable Dynamic Bicycle Model for Discrete-time Control](https://mlsanity.com/paper/2011.09612/) (2020)
- [Optimal Cost Design for Model Predictive Control](https://mlsanity.com/paper/2104.11353/) (2021)

## Abstract (arXiv)

Choosing an appropriate parameter set for the designed controller is critical for the final performance but usually requires a tedious and careful tuning process, which implies a strong need for automatic tuning methods. However, among existing methods, derivative-free ones suffer from poor scalability or low efficiency, while gradient-based ones are often unavailable due to possibly non-differentiable controller structure. To resolve the issues, we tackle the controller tuning problem using a novel derivative-free reinforcement learning (RL) framework, which performs timestep-wise perturbation in parameter space during experience collection and integrates derivative-free policy updates into the advanced actor-critic RL architecture to achieve high versatility and efficiency. To demonstrate the framework's efficacy, we conduct numerical experiments on two concrete examples from autonomous driving, namely, adaptive cruise control with PID controller and trajectory tracking with MPC controller. Experimental results show that the proposed method outperforms popular baselines and highlight its strong potential for controller tuning.