# Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics arXiv: https://arxiv.org/abs/2111.01365 ยท HTML: https://mlsanity.com/paper/2111.01365/ ## Key facts - Cited by 27 papers in the map (data through 14 Oct 2026). - Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. - Builds on 24 of its 40 references that are in the map. - First posted on arXiv 2 Nov 2021. ## Placed in - Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/) ## Builds on (most cited first, 24 of 24) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018) - [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020) - [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018) - [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019) - [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018) - [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018) - [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017) - [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017) - [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019) - [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020) - [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020) - [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020) - [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020) - [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018) - [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021) - [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019) - [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020) - [S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning](https://mlsanity.com/paper/2103.06326/) (2021) - [Learning Compositional Koopman Operators for Model-Based Control](https://mlsanity.com/paper/1910.08264/) (2019) - [Data-driven discovery of Koopman eigenfunctions for control](https://mlsanity.com/paper/1707.01146/) (2017) - [Symmetry-Based Disentangled Representation Learning requires Interaction with Environments](https://mlsanity.com/paper/1904.00243/) (2019) - [Deep Variational Koopman Models: Inferring Koopman Observations for Uncertainty-Aware Dynamics Modeling and Control](https://mlsanity.com/paper/1902.09742/) (2019) - [Koopman Spectrum Nonlinear Regulators and Efficient Online Learning](https://mlsanity.com/paper/2106.15775/) (2021) ## Built on by (most cited first, 27 of 27) - [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024) - [Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL](https://mlsanity.com/paper/2306.04220/) (2023) - [Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2509.06782/) (2025) - [Morphological Symmetries in Robotics](https://mlsanity.com/paper/2402.15552/) (2024) - [Koopman-Assisted Reinforcement Learning](https://mlsanity.com/paper/2403.02290/) (2024) - [Efficient Dynamics Modeling in Interactive Environments with Koopman Theory](https://mlsanity.com/paper/2306.11941/) (2023) - [PyKoopman: A Python Package for Data-Driven Approximation of the Koopman Operator](https://mlsanity.com/paper/2306.12962/) (2023) - [Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning](https://mlsanity.com/paper/2310.18247/) (2023) - [Information Shapes Koopman Representation](https://mlsanity.com/paper/2510.13025/) (2025) - [Koopman Dreamer: Spectrally Constrained Latent Dynamics for Stable World-Model Imagination](https://mlsanity.com/paper/2607.19719/) (2026) - [SiT: Symmetry-Invariant Transformers for Generalisation in Reinforcement Learning](https://mlsanity.com/paper/2406.15025/) (2024) - [Optimal Control of Nonlinear Systems with Unknown Dynamics](https://mlsanity.com/paper/2305.15188/) (2023) - [HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approach](https://mlsanity.com/paper/2306.06329/) (2023) - [Exploiting Symmetry in Dynamics for Model-Based Reinforcement Learning with Asymmetric Rewards](https://mlsanity.com/paper/2403.19024/) (2024) - [Deep deterministic policy gradient with symmetric data augmentation for lateral attitude tracking control of a fixed-wing aircraft](https://mlsanity.com/paper/2407.11077/) (2024) - [Symmetries-enhanced Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.01136/) (2025) - [Koopman-Based Generalization of Deep Reinforcement Learning With Application to Wireless Communications](https://mlsanity.com/paper/2503.02961/) (2025) - [KIPPO: Koopman-Inspired Proximal Policy Optimization](https://mlsanity.com/paper/2505.14566/) (2025) - [Mollified Value Learning](https://mlsanity.com/paper/2602.23280/) (2026) - [DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration](https://mlsanity.com/paper/2603.07545/) (2026) - [Koopman operator theory: fundamentals, control, and applications](https://mlsanity.com/paper/2607.01819/) (2026) - [KEEC: Koopman Embedded Equivariant Control](https://mlsanity.com/paper/2312.01544/) (2023) - [Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixing](https://mlsanity.com/paper/2408.10517/) (2024) - [SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learning](https://mlsanity.com/paper/2502.15512/) (2025) - [Reinforcement Learning Using known Invariances](https://mlsanity.com/paper/2511.03473/) (2025) - [Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixing](https://mlsanity.com/paper/2602.19805/) (2026) - [SKooP: Symmetric Koopman Predictions for Faster and More Generalizable Legged Robot Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2607.11624/) (2026) ## Abstract (arXiv) Offline reinforcement learning leverages large datasets to train policies without interactions with the environment. The learned policies may then be deployed in real-world settings where interactions are costly or dangerous. Current algorithms over-fit to the training dataset and as a consequence perform poorly when deployed to out-of-distribution generalizations of the environment. We aim to address these limitations by learning a Koopman latent representation which allows us to infer symmetries of the system's underlying dynamic. The latter is then utilized to extend the otherwise static offline dataset during training; this constitutes a novel data augmentation framework which reflects the system's dynamic and is thus to be interpreted as an exploration of the environments phase space. To obtain the symmetries we employ Koopman theory in which nonlinear dynamics are represented in terms of a linear operator acting on the space of measurement functions of the system and thus symmetries of the dynamics may be inferred directly. We provide novel theoretical results on the existence and nature of symmetries relevant for control systems such as reinforcement learning settings. Moreover, we empirically evaluate our method on several benchmark offline reinforcement learning tasks and datasets including D4RL, Metaworld and Robosuite and find that by using our framework we consistently improve the state-of-the-art of model-free Q-learning methods.