mlsanityData to 7 Oct 2026

Reinforcement Learning from Delayed Observations via World Models

arXiv 2403.12309 ↗First posted 18 Mar 2024cs.LG cs.AIcited by 13

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

In standard reinforcement learning settings, agents typically assume immediate feedback about the effects of their actions after taking them. However, in practice, this assumption may not hold true due to physical constraints and can significantly impact the performance of learning algorithms. In this paper, we address observation delays in partially observable environments. We propose leveraging world models, which have shown success in integrating past observations and learning dynamics, to handle observation delays. By reducing delayed POMDPs to delayed MDPs with world models, our methods can effectively handle partial observability, where existing approaches achieve sub-optimal performance or degrade quickly as observability decreases. Experiments suggest that one of our methods can outperform a naive model-based approach by up to 250%. Moreover, we evaluate our methods on visual delayed environments, for the first time showcasing delay-aware reinforcement learning continuous control with visual observations.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 17 of its 34 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

17 of 34 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Mastering Diverse Domains through World Models10 Jan
  2. 2021Mastering Atari Games with Limited Data30 Oct
  3. 2020Mastering Atari with Discrete World Models5 Oct
  4. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  5. 2018World Models27 Mar
  6. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models30 Nov

Other references · 11

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  4. 2019Challenges of Real-World Reinforcement Learningcited by 282
  5. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  6. 2020Reinforcement Learning with Random Delayscited by 56
  7. 2020Delay-Aware Model-Based Reinforcement Learning for Continuous Controlcited by 33
  8. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  9. 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 28
  10. 2022Delayed Reinforcement Learning by Imitationcited by 15
  11. 2022Federated Learning Using Variance Reduced Stochastic Gradient for Probabilistically Activated Agentscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Verification-Guided Shielding for Deep Reinforcement Learningcited by 13
  2. 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inferencecited by 7
  3. 2024Tree Search-Based Policy Optimization under Stochastic Execution Delaycited by 2
  4. 2025Model-Based Reinforcement Learning under Random Observation Delayscited by 2
  5. 2024Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learningcited by 1
  6. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimizationcited by 1
  7. 2026Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Surveycited by 1
  8. 2026Latency-Tolerant Cloud-Edge Collaborative Vision-Language-Action Models via Emergent Representational Specializationcited by 1
  9. 2024Inverse Delayed Reinforcement Learningcited by 0
  10. 2025Robust Remote Reinforcement Learning over Unreliable Communication Channels using Homomorphic State Encodingcited by 0
  11. 2026State-Action Inpainting Diffuser for Continuous Control with Delaycited by 0
  12. 2026Delay-Empowered Causal Hierarchical Reinforcement Learningcited by 0
1 more
  1. 2026RAPAC-DP: Response-Aligned Pending-Action Compensation for Diffusion Policies under Delayed Executioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
  2. 2023Continual Visual Reinforcement Learning with A Life-Long World Model
  3. 2025Delay-Aware Diffusion Policy: Bridging the Observation-Execution Gap in Dynamic Tasks
  4. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
  5. 2020Offline Reinforcement Learning from Images with Latent Space Models
  6. 2026Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning
  7. 2023Harnessing Discrete Representations For Continual Reinforcement Learning
  8. 2024Open-World Reinforcement Learning over Long Short-Term Imagination