# Online Algorithms and Policies Using Adaptive and Machine Learning Approaches

arXiv: https://arxiv.org/abs/2105.06577 · HTML: https://mlsanity.com/paper/2105.06577/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 77 references that are in the map.
- First posted on arXiv 13 May 2021.

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Provably Efficient Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/1907.05388/) (2019)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [A Variational Perspective on Accelerated Methods in Optimization](https://mlsanity.com/paper/1603.04245/) (2016)
- [Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator](https://mlsanity.com/paper/1805.09388/) (2018)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Towards Generalization and Simplicity in Continuous Control](https://mlsanity.com/paper/1703.02660/) (2017)
- [Reinforcement Learning under Model Mismatch](https://mlsanity.com/paper/1706.04711/) (2017)
- [Control Contraction Metrics: Convex and Intrinsic Criteria for Nonlinear Feedback Design](https://mlsanity.com/paper/1503.03144/) (2015)
- [Adaptive-Control-Oriented Meta-Learning for Nonlinear Systems](https://mlsanity.com/paper/2103.04490/) (2021)
- [Direct Runge-Kutta Discretization Achieves Acceleration](https://mlsanity.com/paper/1805.00521/) (2018)
- [Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems](https://mlsanity.com/paper/2007.12291/) (2020)
- [Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER](https://mlsanity.com/paper/2012.01399/) (2020)
- [From self-tuning regulators to reinforcement learning and back again](https://mlsanity.com/paper/1906.11392/) (2019)
- [Connections Between Adaptive Control and Optimization in Machine Learning](https://mlsanity.com/paper/1904.05856/) (2019)
- [Online Policies for Real-Time Control Using MRAC-RL](https://mlsanity.com/paper/2103.16551/) (2021)
- [A Stable High-order Tuner for General Convex Functions](https://mlsanity.com/paper/2011.09996/) (2020)
- [Accelerated Learning with Robustness to Adversarial Regressors](https://mlsanity.com/paper/2005.01529/) (2020)

## Built on by (most cited first, 3 of 3)

- [Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches](https://mlsanity.com/paper/2505.03706/) (2025)
- [Extremum-Seeking Action Selection for Accelerating Policy Optimization](https://mlsanity.com/paper/2404.01598/) (2024)
- [Autopilot-Preserving Residual Q-Learning with HJB-Inspired Finite-Action Risk Filtering for Fixed-Wing UAV Command Supervision](https://mlsanity.com/paper/2606.01397/) (2026)

## Abstract (arXiv)

This paper considers the problem of real-time control and learning in dynamic systems subjected to parametric uncertainties. We propose a combination of a Reinforcement Learning (RL) based policy in the outer loop suitably chosen to ensure stability and optimality for the nominal dynamics, together with Adaptive Control (AC) in the inner loop so that in real-time AC contracts the closed-loop dynamics towards a stable trajectory traced out by RL. Two classes of nonlinear dynamic systems are considered, both of which are control-affine. The first class of dynamic systems utilizes equilibrium points %with expansion forms around these points and a Lyapunov approach while second class of nonlinear systems uses contraction theory. AC-RL controllers are proposed for both classes of systems and shown to lead to online policies that guarantee stability using a high-order tuner and accommodate parametric uncertainties and magnitude limits on the input. In addition to establishing a stability guarantee with real-time control, the AC-RL controller is also shown to lead to parameter learning with persistent excitation for the first class of systems. Numerical validations of all algorithms are carried out using a quadrotor landing task on a moving platform.