# An Output Feedback Q-learning Algorithm for Optimal Control of Nonlinear Systems with Koopman Linear Embedding

arXiv: https://arxiv.org/abs/2603.29858 · HTML: https://mlsanity.com/paper/2603.29858/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 8 of 8)

- [Koopman Form of Nonlinear Systems with Inputs](https://mlsanity.com/paper/2207.12132/) (2022)
- [An overview of Koopman-based control: From error bounds to closed-loop guarantees](https://mlsanity.com/paper/2509.02839/) (2025)
- [Efficient Off-Policy Q-Learning for Data-Based Discrete-Time LQR Problems](https://mlsanity.com/paper/2105.07761/) (2021)
- [Willems' Fundamental Lemma for Nonlinear Systems with Koopman Linear Embedding](https://mlsanity.com/paper/2409.16389/) (2024)
- [On the Existence of Koopman Linear Embeddings for Controlled Nonlinear Systems](https://mlsanity.com/paper/2602.14537/) (2026)
- [Notes on data-driven output-feedback control of linear MIMO systems](https://mlsanity.com/paper/2311.17484/) (2023)
- [On the Exponential Stability of Koopman Model Predictive Control](https://mlsanity.com/paper/2511.02008/) (2025)
- [An efficient data-based off-policy Q-learning algorithm for optimal output feedback control of linear systems](https://mlsanity.com/paper/2312.03451/) (2023)

## Abstract (arXiv)

In the reinforcement learning literature, strong theoretical guarantees have been obtained for algorithms applicable to LTI systems. However, in the nonlinear case only weaker results have been obtained for algorithms that mostly rely on the use of function approximation strategies like, for example, neural networks. In this paper, we study the applicability of a known output-feedback Q-learning algorithm to the class of nonlinear systems that admit a Koopman linear embedding. This algorithm uses only input-output data, and no knowledge of either the system model or the Koopman lifting functions is required. Moreover, no function approximation techniques are used, and the same theoretical guarantees as for LTI systems are preserved. Furthermore, we analyze the performance of the algorithm when the Koopman linear embedding is only an approximation of the real nonlinear system. A simulation example verifies the applicability of this method.