# Optimisation of Structured Neural Controller Based on Continuous-Time Policy Gradient

arXiv: https://arxiv.org/abs/2201.06262 · HTML: https://mlsanity.com/paper/2201.06262/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 19 of 19)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [A Differentiable Programming System to Bridge Machine Learning and Scientific Computing](https://mlsanity.com/paper/1907.07587/) (2019)
- [Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework](https://mlsanity.com/paper/1912.12970/) (2019)
- [Don't Unroll Adjoint: Differentiating SSA-Form Programs](https://mlsanity.com/paper/1810.07951/) (2018)
- [Continuous-Time Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.04764/) (2021)
- [Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls](https://mlsanity.com/paper/2010.14087/) (2020)
- [An Online Learning Approach to Model Predictive Control](https://mlsanity.com/paper/1902.08967/) (2019)
- [Stein Variational Model Predictive Control](https://mlsanity.com/paper/2011.07641/) (2020)
- [Real-time optimal control via Deep Neural Networks: study on landing problems](https://mlsanity.com/paper/1610.08668/) (2016)
- [Safe Pontryagin Differentiable Programming](https://mlsanity.com/paper/2105.14937/) (2021)
- [Youla-REN: Learning Nonlinear Feedback Policies with Robust Stability Guarantees](https://mlsanity.com/paper/2112.01253/) (2021)
- [Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time](https://mlsanity.com/paper/1912.10697/) (2019)
- [Adaptive Meta-Learning for Identification of Rover-Terrain Dynamics](https://mlsanity.com/paper/2009.10191/) (2020)
- [Faster Policy Learning with Continuous-Time Gradients](https://mlsanity.com/paper/2012.06684/) (2020)
- [Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control](https://mlsanity.com/paper/2009.13097/) (2020)
- [A Domain-Knowledge-Aided Deep Reinforcement Learning Approach for Flight Control Design](https://mlsanity.com/paper/1908.06884/) (2019)

## Built on by (most cited first, 1 of 1)

- [Incremental Correction in Dynamic Systems Modelled with Neural Networks for Constraint Satisfaction](https://mlsanity.com/paper/2209.03698/) (2022)

## Abstract (arXiv)

This study presents a policy optimisation framework for structured nonlinear control of continuous-time (deterministic) dynamic systems. The proposed approach prescribes a structure for the controller based on relevant scientific knowledge (such as Lyapunov stability theory or domain experiences) while considering the tunable elements inside the given structure as the point of parametrisation with neural networks. To optimise a cost represented as a function of the neural network weights, the proposed approach utilises the continuous-time policy gradient method based on adjoint sensitivity analysis as a means for correct and performant computation of cost gradient. This enables combining the stability, robustness, and physical interpretability of an analytically-derived structure for the feedback controller with the representational flexibility and optimised resulting performance provided by machine learning techniques. Such a hybrid paradigm for fixed-structure control synthesis is particularly useful for optimising adaptive nonlinear controllers to achieve improved performance in online operation, an area where the existing theory prevails the design of structure while lacking clear analytical understandings about tuning of the gains and the uncertainty model basis functions that govern the performance characteristics. Numerical experiments on aerospace applications illustrate the utility of the structured nonlinear controller optimisation framework.