# WorldKV: Efficient World Memory with World Retrieval and Compression

arXiv: https://arxiv.org/abs/2605.22718 · HTML: https://mlsanity.com/paper/2605.22718/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 21 May 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)
- Efficient LLM inference and serving / [Fast autoregressive visual generation](https://mlsanity.com/inference/fast-autoregressive-visual-generation/)

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control](https://mlsanity.com/paper/2503.03751/) (2025)
- [MAGI-1: Autoregressive Video Generation at Scale](https://mlsanity.com/paper/2505.13211/) (2025)
- [LongLive: Real-time Interactive Long Video Generation](https://mlsanity.com/paper/2509.22622/) (2025)
- [Rolling Forcing: Autoregressive Long Video Diffusion in Real Time](https://mlsanity.com/paper/2509.25161/) (2025)
- [SkyReels-V2: Infinite-length Film Generative Model](https://mlsanity.com/paper/2504.13074/) (2025)
- [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025)
- [Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation](https://mlsanity.com/paper/2602.02214/) (2026)
- [WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling](https://mlsanity.com/paper/2512.14614/) (2025)
- [WorldMem: Long-term Consistent World Simulation with Memory](https://mlsanity.com/paper/2504.12369/) (2025)
- [Advancing Open-source World Models](https://mlsanity.com/paper/2601.20540/) (2026)
- [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026)
- [VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory](https://mlsanity.com/paper/2506.18903/) (2025)
- [RELIC: Interactive Video World Model with Long-Horizon Memory](https://mlsanity.com/paper/2512.04040/) (2025)
- [Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression](https://mlsanity.com/paper/2512.05081/) (2025)
- [Yume-1.5: A Text-Controlled Interactive World Generation Model](https://mlsanity.com/paper/2512.22096/) (2025)
- [R-KV: Redundancy-aware KV Cache Compression for Reasoning Models](https://mlsanity.com/paper/2505.24133/) (2025)
- [Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft](https://mlsanity.com/paper/2510.03198/) (2025)
- [Solaris: Building a Multiplayer Video World Model in Minecraft](https://mlsanity.com/paper/2602.22208/) (2026)
- [Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs](https://mlsanity.com/paper/2503.00979/) (2025)
- [Grounding World Simulation Models in a Real-World Metropolis](https://mlsanity.com/paper/2603.15583/) (2026)

## Built on by (most cited first, 15 of 15)

- [Magpie: Real-Time World Renderer for Interactive Games](https://mlsanity.com/paper/2608.27168/) (2026)
- [ConsistWorld: Evidence Routing for Consistent Multi-Agent World Models](https://mlsanity.com/paper/2609.22641/) (2026)
- [Can Video World Models Track Unobserved World States?](https://mlsanity.com/paper/2608.30692/) (2026)
- [WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory](https://mlsanity.com/paper/2609.24984/) (2026)
- [Precise Editing and Flexible Referencing for Interactable Worlds](https://mlsanity.com/paper/2609.34470/) (2026)
- [Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Models](https://mlsanity.com/paper/2607.10389/) (2026)
- [World in World: Explore the World with World Models](https://mlsanity.com/paper/2609.11548/) (2026)
- [Revision, Not Restart: Revisable Visual Plans for Closed-Loop World-Action Models](https://mlsanity.com/paper/2609.35439/) (2026)
- [Compress to Remember: Learning Compact Memory via On-Policy Distillation for Long Video Generation](https://mlsanity.com/paper/2609.36364/) (2026)
- [Honeycomb: Constant-Size Scene Memory Representation for Video World Models](https://mlsanity.com/paper/2609.37690/) (2026)
- [DeCoPrune: Efficient KV-Cache Pruning for Autoregressive Video Diffusion via Denoising Consistency](https://mlsanity.com/paper/2609.39096/) (2026)
- [LOCI: Spatial Linear Memory for Streaming World Models](https://mlsanity.com/paper/2609.40222/) (2026)
- [Memorizon: Training World Models Beyond Their Context Window](https://mlsanity.com/paper/2610.00544/) (2026)
- [World Observer: Joint Actor-Observer Generation for Persistent World Modeling](https://mlsanity.com/paper/2610.02162/) (2026)
- [4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling](https://mlsanity.com/paper/2610.17625/) (2026)

## Abstract (arXiv)

Autoregressive video diffusion models have enabled real-time, action-conditioned world generation. However, sustaining a persistent world, where revisiting a previously seen viewpoint yields consistent content, remains an open problem. Full KV-cache attention preserves this consistency but breaks real-time constraints: memory footprint and attention cost grow linearly with rollout length. Sliding window inference restores throughput but discards long-term consistency. We propose WorldKV, a training-free framework with two components: World Retrieval and World Compression. World Retrieval stores evicted KV-cache chunks in GPU/CPU memory and selectively retrieves scene-relevant chunks via camera/ action correspondence, inserting them back into the native attention window without re-encoding. World Compression prunes redundant tokens within each chunk via key-key similarity to an anchor frame, halving per-chunk storage to fit 2x more history under a fixed budget. On Matrix-Game-2.0 and LingBot- World-Fast, WorldKV matches or exceeds full-KV memory fidelity at roughly 2x the throughput, and is competitive with memory-trained baselines without any fine-tuning. Project Page: https://cvlab-kaist.github.io/WorldKV/