mlsanityData to 14 Oct 2026

Deep SPI: Safe Policy Improvement via World Models

arXiv 2510.12312 ↗First posted 14 Oct 2025cs.LG cs.AIcited by 3

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

Safe policy improvement (SPI) offers theoretical control over policy updates, yet existing guarantees largely concern offline, tabular reinforcement learning (RL). We study SPI in general online settings, when combined with world model and representation learning. We develop a theoretical framework showing that restricting policy updates to a well-defined neighborhood of the current policy ensures monotonic improvement and convergence. This analysis links transition and reward prediction losses to representation quality, yielding online, "deep" analogues of classical SPI theorems from the offline RL literature. Building on these results, we introduce DeepSPI, a principled on-policy algorithm that couples local transition and reward losses with regularised policy updates. On the ALE-57 benchmark, DeepSPI matches or exceeds strong baselines, including PPO and DeepMDPs, while retaining theoretical guarantees.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 39 of its 66 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

39 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2018World Models27 Mar

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2017Improved Training of Wasserstein GANscited by 2,235
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  7. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  9. 2019A Theory of Regularized Markov Decision Processescited by 286
  10. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  11. 2018Sorting out Lipschitz function approximationcited by 213
  12. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
25 more
  1. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  2. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  3. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  4. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  5. 2024Simplifying Deep Temporal Difference Learningcited by 78
  6. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  7. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  8. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 48
  9. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  10. 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39
  11. 2018Combined Reinforcement Learning via Abstract Representationscited by 37
  12. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  13. 2021Stability Verification in Stochastic Control Systems via Neural Network Supermartingalescited by 26
  14. 2020Steady State Analysis of Episodic Reinforcement Learningcited by 21
  15. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meetcited by 18
  16. 2019Safe Policy Improvement with an Estimated Baseline Policycited by 17
  17. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  18. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 13
  19. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  20. 2021Distillation of RL Policies with Formal Guarantees via Variational Abstraction of Markov Decision Processes (Technical Report)cited by 9
  21. 2023The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Modelscited by 9
  22. 2023More for Less: Safe Policy Improvement With Stronger Performance Guaranteescited by 9
  23. 2023Model-based Offline Reinforcement Learning with Local Misspecificationcited by 5
  24. 2023A Tale of Sampling and Estimation in Discounted Reinforcement Learningcited by 4
  25. 2024Composing Reinforcement Learning Policies, with Formal Guaranteescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Medical world models in healthcare: foundations, applications, and challenges for trustworthy clinical translationcited by 4
  2. 2026Active Inference as a Convex Markov Decision Processcited by 1
  3. 2026Bellman-Certified Rounding for Sparse Policy Deployment in MDPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  2. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  3. 2025Data-Efficient Safe Policy Improvement Using Parametric Structure
  4. 2023Safe Policy Improvement for POMDPs via Finite-State Controllers
  5. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
  6. 2024Decision-Point Guided Safe Policy Improvement
  7. 2022Safe Policy Improvement Approaches and their Limitations
  8. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning