mlsanityData to 14 Oct 2026

WorldMem: Long-term Consistent World Simulation with Memory

arXiv 2504.12369 ↗First posted 16 Apr 2025cs.CVcited by 168

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing a memory attention mechanism that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.

Key facts. Cited by 168 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 49 of its 62 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

49 of 62 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025GameFactory: Creating New Games with Generative Interactive Videos14 Jan
  2. 2024Diffusion Models Are Real-Time Game Engines27 Aug
  3. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  4. 2023Learning Interactive Real-World Simulators9 Oct
  5. 2022Scalable Diffusion Models with Transformers19 Dec
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  6. 2018World Modelscited by 1,513
  7. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  8. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  9. 2020Mastering Atari with Discrete World Modelscited by 1,007
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  11. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  12. 2023ModelScope Text-to-Video Technical Reportcited by 658
31 more
  1. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  2. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  3. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  4. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  5. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  7. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  8. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  9. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  10. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  11. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  12. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
  13. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  14. 20243D Reconstruction with Spatial Memorycited by 277
  15. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 227
  16. 2019Understanding and Improving Layer Normalizationcited by 217
  17. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 202
  18. 2025History-Guided Video Diffusioncited by 171
  19. 2023WonderJourney: Going from Anywhere to Everywherecited by 158
  20. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 143
  21. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 137
  22. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 118
  23. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 99
  24. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80
  25. 2022Make-A-Story: Visual Memory Conditioned Consistent Story Generationcited by 76
  26. 2024Trajectory Attention for Fine-grained Video Motion Controlcited by 63
  27. 2025GenFusion: Closing the Loop between Reconstruction and Generation via Videoscited by 41
  28. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generationcited by 38
  29. 2024SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generationcited by 28
  30. 2025Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Frameworkcited by 20
  31. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 44; 2026: 124.

168 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 170
  3. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  4. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 100
  5. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
  6. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 96
  7. 2025Mixture of Contexts for Long Video Generationcited by 95
  8. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 61
  9. 2026Context Forcing: Consistent Autoregressive Video Generation with Long Contextcited by 59
  10. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraftcited by 52
  11. 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
  12. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 47
38 more
  1. 2025MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narrativescited by 45
  2. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 44
  3. 2025VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memorycited by 42
  4. 2025RoboScape: Physics-informed Embodied World Modelcited by 40
  5. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 40
  6. 2025StoryMem: Multi-shot Long Video Storytelling with Memorycited by 40
  7. 2026Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memorycited by 40
  8. 2025VRAG: Learning World Models for Interactive Video Generationcited by 38
  9. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  10. 2026MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokenscited by 27
  11. 2026MosaicMem: Hybrid Spatial Memory for Controllable Video World Modelscited by 27
  12. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 26
  13. 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Modelscited by 26
  14. 20253D Scene Generation: A Surveycited by 24
  15. 2025PanoWan: Lifting Diffusion Video Generation Models to 360deg with Latitude/Longitude-aware Mechanismscited by 23
  16. 2026Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantizationcited by 23
  17. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 23
  18. 2025PlayerOne: Egocentric World Simulatorcited by 20
  19. 2025EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memorycited by 20
  20. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 19
  21. 2025LoViC: Efficient Long Video Generation with Context Compressioncited by 19
  22. 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 18
  23. 2026MBench: A Comprehensive Benchmark on Memory Capability for Video World Modelscited by 18
  24. 2026Latent Spatial Memory for Video World Modelscited by 17
  25. 2026LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generationcited by 16
  26. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  27. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 15
  28. 2026WorldKV: Efficient World Memory with World Retrieval and Compressioncited by 15
  29. 2026Gamma-World: Generative Multi-Agent World Modeling Beyond Two Playerscited by 15
  30. 2026Addressable Memory for Video World Modelscited by 15
  31. 2025Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresightcited by 14
  32. 2026Code World Model: Coding Agent as World Braincited by 14
  33. 2025Reinforcement Learning with Inverse Rewards for World Model Post-trainingcited by 13
  34. 2026Beyond Pixel Histories: World Models with Persistent 3D Statecited by 13
  35. 2026Geometry-Aware Implicit Memory for Video World Modelscited by 13
  36. 2026EchoWM: Open and Enterable Omnimodal World Modelscited by 13
  37. 2026StableWorld: Towards Stable and Consistent Long Interactive Video Generationcited by 12
  38. 2026Mode Seeking meets Mean Seeking for Fast Long Video Generationcited by 12

50 most cited of 168 All 168 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Video World Models with Long-term Spatial Memory
  2. 2026WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models
  3. 2026LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models
  4. 2026MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
  5. 2025WorldPack: Dynamic Frame Compression for Long-context Video World Modeling
  6. 2026Perceptual 3D Simulation With Physical World Modeling
  7. 2026WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
  8. 2025Learning 3D Persistent Embodied World Models