mlsanityData to 14 Oct 2026

Learning Abstract World Model for Value-preserving Planning with Options

arXiv 2406.15850 ↗First posted 22 Jun 2024cs.LG cs.AIcited by 4

On the map: World models → Latent predictive world models

Abstract (arXiv)

General-purpose agents require fine-grained controls and rich sensory inputs to perform a wide range of tasks. However, this complexity often leads to intractable decision-making. Traditionally, agents are provided with task-specific action and observation spaces to mitigate this challenge, but this reduces autonomy. Instead, agents must be capable of building state-action spaces at the correct abstraction level from their sensorimotor experiences. We leverage the structure of a given set of temporally-extended actions to learn abstract Markov decision processes (MDPs) that operate at a higher level of temporal and state granularity. We characterize state abstractions necessary to ensure that planning with these skills, by simulating trajectories in the abstract MDP, results in policies with bounded value loss in the original MDP. We evaluate our approach in goal-based navigation environments that require continuous abstract states to plan successfully and show that abstract model learning improves the sample efficiency of planning and learning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 34 of its 74 references that are in the map. First posted on arXiv 22 Jun 2024.

Builds on

34 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Deep Hierarchical Planning from Pixels8 Jun
  2. 2022Temporal Difference Learning for Model Predictive Control9 Mar
  3. 2018Learning Latent Dynamics for Planning from Pixels12 Nov

Other references · 31

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2020Mastering Atari with Discrete World Modelscited by 1,007
  9. 2016Deep Variational Information Bottleneckcited by 930
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  11. 2018MINE: Mutual Information Neural Estimationcited by 683
  12. 2019On Variational Bounds of Mutual Informationcited by 559
19 more
  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  2. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  3. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  4. 2015Deep Kalman Filterscited by 166
  5. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 94
  6. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  7. 2017Near Optimal Behavior via Approximate State Abstractioncited by 66
  8. 2017Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundarycited by 58
  9. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 55
  10. 2022Skill-based Model-based Reinforcement Learningcited by 50
  11. 2018Temporal Difference Variational Auto-Encodercited by 49
  12. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  13. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 45
  14. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 44
  15. 2019Unsupervised Grounding of Plannable First-Order Logic Representation from Imagescited by 24
  16. 2021Classical Planning in Deep Latent Spacecited by 22
  17. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  18. 2019Learning Portable Representations for High-Level Planningcited by 11
  19. 2023Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domainscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024SkiLD: Unsupervised Skill Discovery Guided by Factor Interactionscited by 9
  2. 2026Compositional Planning with Jumpy World Modelscited by 7
  3. 2025From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learningcited by 3
  4. 2025Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SLAP: Shortcut Learning for Abstract Planning
  2. 2025Learning with Expert Abstractions for Efficient Multi-Task Continuous Control
  3. 2019Planning with Abstract Learned Models While Learning Transferable Subtasks
  4. 2024Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning
  5. 2026Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning
  6. 2024Reconciling Spatial and Temporal Abstractions for Goal Representation
  7. 2026When to Plan: Learning to Select Between Reactive Control and Deliberative Planning
  8. 2026TodoEvolve: Learning to Architect Agent Planning Systems