mlsanityData to 7 Oct 2026

Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models

arXiv 2609.34677 ↗First posted 28 Sep 2026cs.LG cs.AI cs.CVcited by 0

On the map: World models → Latent predictive world models also fits: Video-generation world models and simulators

Abstract (arXiv)

World models predict future observations from current experience and actions, yet prediction can depend on observations seen far in the past. Episodic memory preserves past observations for later recall; however, as memory accumulates, it raises a fundamental question: which memories are useful for the current prediction, and which available retrieval cues should be trusted to find them? This is challenging because fixed criteria based on recency, pose overlap, or visual similarity can be unreliable across environments and queries. We propose Future-Aware Recall (FAR), a framework that learns episodic recall from future-aware predictive supervision and adaptive multi-cue scoring. During training, FAR measures predictive utility by the conditional log-likelihood of the realized future given recalled context, approximated by negative diffusion prediction loss, and uses it to train a retriever that remains future-blind at inference. The retriever learns cue-specific relevance and automatically determines which available retrieval cues, such as time, pose, vision, and audio, to trust for each query when selecting memories. Across three complementary settings, FAR outperforms hand-designed recall even with the same retrieval cues, automatically adapts which available cues to trust, and recalls the right history as the world changes. Together, these results establish FAR as a flexible, principled approach to episodic memory access in world models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 39 of its 42 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

39 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model5 Mar
  2. 2018Learning Latent Dynamics for Planning from Pixels12 Nov

Other references · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  9. 2021Variational Diffusion Modelscited by 1,145
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
25 more
  1. 2024Genie: Generative Interactive Environmentscited by 706
  2. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  3. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  4. 2024Navigation World Modelscited by 262
  5. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  6. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  7. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  8. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  9. 2025Video World Models with Long-term Spatial Memorycited by 124
  10. 2025Mixture of Contexts for Long Video Generationcited by 90
  11. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 86
  12. 2025The Principles of Diffusion Modelscited by 70
  13. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50
  14. 2025VRAG: Learning World Models for Interactive Video Generationcited by 37
  15. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 33
  16. 2026LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generationcited by 12
  17. 2026Beyond Pixel Histories: World Models with Persistent 3D Statecited by 11
  18. 2026EgoSim: Egocentric World Simulator for Embodied Interaction Generationcited by 11
  19. 2026Grounding World Simulation Models in a Real-World Metropoliscited by 10
  20. 2026MemLearner: Learning to Query Context memory for Video World Modelscited by 8
  21. 2025LoopNav: Benchmarking Spatial Consistency in World Modelscited by 7
  22. 2026Compression and Retrieval: Implicit Memory Retrieval for Video World Modelscited by 6
  23. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
  24. 2026I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generationcited by 2
  25. 2026MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FlashBack: Knowing When to Remember in Streaming Vision-Language Models
  2. 2026Learning What to Remember: A Cognitively Grounded Multi-Factor Value Model for Agentic Memory
  3. 2026RRM: Experience-Driven Reflective Retrieval Memory for Long-Horizon Multimodal Reasoning
  4. 2026Composition of Memory Experts for Diffusion World Models
  5. 2026Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes
  6. 2025Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigation
  7. 2026Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution
  8. 2026Memory by Design: Probabilistic Sequence Layers