mlsanityData to 7 Oct 2026

Operator World Models for Reinforcement Learning

arXiv 2406.19861 ↗First posted 28 Jun 2024cs.LG math.OC stat.MLcited by 7

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Policy Mirror Descent (PMD) is a powerful and theoretically sound methodology for sequential decision-making. However, it is not directly applicable to Reinforcement Learning (RL) due to the inaccessibility of explicit action-value functions. We address this challenge by introducing a novel approach based on learning a world model of the environment using conditional mean embeddings. Leveraging tools from operator theory we derive a closed-form expression of the action-value function in terms of the world model via simple matrix operations. Combining these estimators with PMD leads to POWR, a new RL algorithm for which we prove convergence rates to the global optimum. Preliminary experiments in finite and infinite state settings support the effectiveness of our method

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 22 of its 45 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

22 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Mastering Diverse Domains through World Models10 Jan
  2. 2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2014Convex Optimization: Algorithms and Complexitycited by 569
  5. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 360
  6. 2019A Theory of Regularized Markov Decision Processescited by 279
  7. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  8. 2022On the Convergence Rates of Policy Gradient Methodscited by 115
  9. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
  10. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  11. 2015Less is More: Nystr"om Computational Regularizationcited by 89
  12. 2022Learning Dynamical Systems via Koopman Operator Regression in Reproducing Kernel Hilbert Spacescited by 66
8 more
  1. 2022Optimal Rates for Regularized Conditional Mean Embedding Learningcited by 50
  2. 2020A General Framework for Consistent Structured Prediction with Implicit Loss Embeddingscited by 35
  3. 2023Sharp Spectral Rates for Koopman Operator Learningcited by 35
  4. 2019Sinkhorn Barycenters with Free Support via Frank-Wolfe Algorithmcited by 27
  5. 2023Estimating Koopman operators with sketching to provably learn large scale dynamical systemscited by 15
  6. 2024Koopman-Assisted Reinforcement Learningcited by 11
  7. 2023Optimistic Planning by Regularized Dynamic Programmingcited by 8
  8. 2023A randomized algorithm to solve reduced rank operator regressioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systemscited by 7
  2. 2026TRAP: Tail-aware Ranking Attack for World-Model Planningcited by 4
  3. 2026HaM-World: Soft-Hamiltonian World Models with Selective Memory for Planningcited by 1
  4. 2026Conditional KRR: Injecting Unpenalized Features into Kernel Methods with Applications to Kernel Thresholdingcited by 0
  5. 2026Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximizationcited by 0
  6. 2026Learning to control switching nonlinear systems with Koopman operator regressioncited by 0
  7. 2026Learning Conditional Expectation Operators via Functional Newton Updatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Approximation of Log-Partition Function in Policy Mirror Descent Induces Implicit Regularization for LLM Post-Training
  2. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
  3. 2022Discovered Policy Optimisation
  4. 2024PWM: Policy Learning with Multi-Task World Models
  5. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
  6. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
  7. 2024MoMA: Model-based Mirror Ascent for Offline Reinforcement Learning
  8. 2026Bellman Policy Optimization