# Optimal Control of Nonlinear Systems with Unknown Dynamics

arXiv: https://arxiv.org/abs/2305.15188 · HTML: https://mlsanity.com/paper/2305.15188/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation](https://mlsanity.com/paper/1806.02450/) (2018)
- [Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control](https://mlsanity.com/paper/1611.03537/) (2016)
- [Dynamic mode decomposition with control](https://mlsanity.com/paper/1409.6358/) (2014)
- [On Convergence of Extended Dynamic Mode Decomposition to the Koopman Operator](https://mlsanity.com/paper/1703.04680/) (2017)
- [Deep Dynamical Modeling and Control of Unsteady Fluid Flows](https://mlsanity.com/paper/1805.07472/) (2018)
- [Generalizing Koopman Theory to allow for inputs and control](https://mlsanity.com/paper/1602.07647/) (2016)
- [Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning](https://mlsanity.com/paper/1703.03078/) (2017)
- [Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics](https://mlsanity.com/paper/2111.01365/) (2021)
- [Deep Learning of Koopman Representation for Control](https://mlsanity.com/paper/2010.07546/) (2020)
- [Linear identification of nonlinear systems: A lifting technique based on the Koopman operator](https://mlsanity.com/paper/1605.04457/) (2016)
- [Deep Koopman Learning of Nonlinear Time-Varying Systems](https://mlsanity.com/paper/2210.06272/) (2022)
- [Towards Global Optimal Control via Koopman Lifts](https://mlsanity.com/paper/2003.01265/) (2020)

## Built on by (most cited first, 1 of 1)

- [Online Control-Informed Learning](https://mlsanity.com/paper/2410.03924/) (2024)

## Abstract (arXiv)

This paper presents a data-driven method to find a closed-loop optimal controller, which minimizes a specified infinite-horizon cost function for systems with unknown dynamics. Suppose the closed-loop optimal controller can be parameterized by a given class of functions, hereafter referred to as the policy. The proposed method introduces a novel gradient estimation framework, which approximates the gradient of the cost function with respect to the policy parameters via integrating the Koopman operator with the classical concept of actor-critic. This enables the policy parameters to be tuned iteratively using gradient descent to achieve an optimal controller, leveraging the linearity of the Koopman operator. The convergence analysis of the proposed framework is provided. The control performance of the proposed method is evaluated through simulations compared with classical optimal control methods that usually assume the dynamics are known.