mlsanityData to 14 Oct 2026

Automatic feature identification in least-squares policy iteration using the Koopman operator framework

arXiv 2603.26464 ↗First posted 27 Mar 2026cs.LG math.DScited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In this paper, we present a Koopman autoencoder-based least-squares policy iteration (KAE-LSPI) algorithm in reinforcement learning (RL). The KAE-LSPI algorithm is based on reformulating the so-called least-squares fixed-point approximation method in terms of extended dynamic mode decomposition (EDMD), thereby enabling automatic feature learning via the Koopman autoencoder (KAE) framework. The approach is motivated by the lack of a systematic choice of features or kernels in linear RL techniques. We compare the KAE-LSPI algorithm with two previous works, the classical least-squares policy iteration (LSPI) and the kernel-based least-squares policy iteration (KLSPI), using stochastic chain walk and inverted pendulum control problems as examples. Unlike previous works, no features or kernels need to be fixed a priori in our approach. Empirical results show the number of features learned by the KAE technique remains reasonable compared to those fixed in the classical LSPI algorithm. The convergence to an optimal or a near-optimal policy is also comparable to the other two methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 5 of its 13 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

5 of 13 references.

References · 5

  1. 2017Deep learning for universal linear embeddings of nonlinear dynamicscited by 494
  2. 2017Linearly-Recurrent Autoencoder Networks for Learning Dynamicscited by 145
  3. 2020Forecasting Sequential Data using Consistent Koopman Autoencoderscited by 115
  4. 2021Finite-data error bounds for Koopman-based prediction and controlcited by 26
  5. 2023Beyond expectations: Residual Dynamic Mode Decomposition and Variance for Stochastic Dynamical Systemscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KIPPO: Koopman-Inspired Proximal Policy Optimization
  2. 2026Efficient Reinforcement Learning using Linear Koopman Dynamics for Nonlinear Robotic Systems
  3. 2024Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator
  4. 2024Koopman-Assisted Reinforcement Learning
  5. 2026Interpreting Reinforcement Learning Model Behavior via Koopman with Control
  6. 2025Leveraging Reinforcement Learning and Koopman Theory for Enhanced Model Predictive Control Performance
  7. 2026An Output Feedback Q-learning Algorithm for Optimal Control of Nonlinear Systems with Koopman Linear Embedding
  8. 2024Model Predictive Control of Nonlinear Dynamics Using Online Adaptive Koopman Operators ↗