# Equipping Black-Box Policies with Model-Based Advice for Stable Nonlinear Control

arXiv: https://arxiv.org/abs/2206.01341 · HTML: https://mlsanity.com/paper/2206.01341/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 2 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [Competitive caching with machine learned advice](https://mlsanity.com/paper/1802.05399/) (2018)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](https://mlsanity.com/paper/1803.00101/) (2018)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Temporal Difference Models: Model-Free Deep RL for Model-Based Control](https://mlsanity.com/paper/1802.09081/) (2018)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Optimal Robustness-Consistency Trade-offs for Learning-Augmented Online Algorithms](https://mlsanity.com/paper/2010.11443/) (2020)
- [Learning Model Predictive Control for iterative tasks. A Data-Driven Control Framework](https://mlsanity.com/paper/1609.01387/) (2016)
- [Control Regularization for Reduced Variance Reinforcement Learning](https://mlsanity.com/paper/1905.05380/) (2019)
- [Unsupervised Curricula for Visual Meta-Reinforcement Learning](https://mlsanity.com/paper/1912.04226/) (2019)
- [Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation](https://mlsanity.com/paper/1911.03845/) (2019)
- [Stability-certified reinforcement learning: A control-theoretic perspective](https://mlsanity.com/paper/1810.11505/) (2018)
- [Robustness and Consistency in Linear Quadratic Control with Untrusted Predictions](https://mlsanity.com/paper/2106.09659/) (2021)
- [Competitive Control with Delayed Imperfect Information](https://mlsanity.com/paper/2010.11637/) (2020)
- [On the Equivalence of Youla, System-level and Input-output Parameterizations](https://mlsanity.com/paper/1907.06256/) (2019)

## Built on by (most cited first, 6 of 6)

- [Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings](https://mlsanity.com/paper/2305.16614/) (2023)
- [Robust Learning for Smoothed Online Convex Optimization with Feedback Delay](https://mlsanity.com/paper/2310.20098/) (2023)
- [Simplex-enabled Safe Continual Learning Machine](https://mlsanity.com/paper/2409.05898/) (2024)
- [Physics-model-guided Worst-case Sampling for Safe Reinforcement Learning](https://mlsanity.com/paper/2412.13224/) (2024)
- [Real-DRL: Teach and Learn in Reality](https://mlsanity.com/paper/2511.00112/) (2025)
- [Cognitively-Grounded On-Device Runtime Learning for Ground Robots in Unknown Physical Environments](https://mlsanity.com/paper/2608.28677/) (2026)

## Abstract (arXiv)

Machine-learned black-box policies are ubiquitous for nonlinear control problems. Meanwhile, crude model information is often available for these problems from, e.g., linear approximations of nonlinear dynamics. We study the problem of equipping a black-box control policy with model-based advice for nonlinear control on a single trajectory. We first show a general negative result that a naive convex combination of a black-box policy and a linear model-based policy can lead to instability, even if the two policies are both stabilizing. We then propose an adaptive λ-confident policy, with a coefficient λ indicating the confidence in a black-box policy, and prove its stability. With bounded nonlinearity, in addition, we show that the adaptive λ-confident policy achieves a bounded competitive ratio when a black-box policy is near-optimal. Finally, we propose an online learning approach to implement the adaptive λ-confident policy and verify its efficacy in case studies about the CartPole problem and a real-world electric vehicle (EV) charging problem with data bias due to COVID-19.