# Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusion

arXiv: https://arxiv.org/abs/2601.21896 · HTML: https://mlsanity.com/paper/2601.21896/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 45 of its 50 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)
- Efficient LLM inference and serving / [Fast autoregressive visual generation](https://mlsanity.com/inference/fast-autoregressive-visual-generation/)

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [Pyramidal Flow Matching for Efficient Video Generative Modeling](https://mlsanity.com/paper/2410.05954/) (2024)
- [MAGI-1: Autoregressive Video Generation at Scale](https://mlsanity.com/paper/2505.13211/) (2025)
- [LongLive: Real-time Interactive Long Video Generation](https://mlsanity.com/paper/2509.22622/) (2025)
- [Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://mlsanity.com/paper/2407.11550/) (2024)
- [Rolling Forcing: Autoregressive Long Video Diffusion in Real Time](https://mlsanity.com/paper/2509.25161/) (2025)
- [SkyReels-V2: Infinite-length Film Generative Model](https://mlsanity.com/paper/2504.13074/) (2025)
- [Autoregressive Video Generation without Vector Quantization](https://mlsanity.com/paper/2412.14169/) (2024)
- [VideoBooth: Diffusion-based Video Generation with Image Prompts](https://mlsanity.com/paper/2312.00777/) (2023)
- [RazorAttention: Efficient KV Cache Compression Through Retrieval Heads](https://mlsanity.com/paper/2407.15891/) (2024)
- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)

## Built on by (most cited first, 16 of 16)

- [Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation](https://mlsanity.com/paper/2603.21366/) (2026)
- [AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories](https://mlsanity.com/paper/2602.14941/) (2026)
- [Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2605.09681/) (2026)
- [Addressable Memory for Video World Models](https://mlsanity.com/paper/2608.07408/) (2026)
- [Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation](https://mlsanity.com/paper/2605.30083/) (2026)
- [RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO](https://mlsanity.com/paper/2605.15190/) (2026)
- [Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion](https://mlsanity.com/paper/2605.16579/) (2026)
- [OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation](https://mlsanity.com/paper/2605.30519/) (2026)
- [DensityKV: Density-Guided KV Cache Compression for Long Video Generation](https://mlsanity.com/paper/2608.27922/) (2026)
- [Inverting the Streaming-Diffusion Bottleneck: Video-Rate MLLM-Conditioned Edit Diffusion on a Consumer GPU](https://mlsanity.com/paper/2606.05981/) (2026)
- [Recency Forcing: Bridging the Long-Horizon Gap in Autoregressive Video Generation](https://mlsanity.com/paper/2609.19729/) (2026)
- [FrameMorrow: Future-guided Frame Selection with Prospective Tokens for Long-Horizon Video Generation](https://mlsanity.com/paper/2609.38839/) (2026)
- [DeCoPrune: Efficient KV-Cache Pruning for Autoregressive Video Diffusion via Denoising Consistency](https://mlsanity.com/paper/2609.39096/) (2026)
- [MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation](https://mlsanity.com/paper/2610.02153/) (2026)
- [FrameFold: Response-Guided KV Cache Consolidation for Autoregressive Long Video Generation](https://mlsanity.com/paper/2610.14568/) (2026)
- [TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generation](https://mlsanity.com/paper/2610.15324/) (2026)

## Abstract (arXiv)

Video generation is pivotal to digital media creation, and recent advances in autoregressive video generation have markedly enhanced the efficiency of real-time video synthesis. However, existing approaches generally rely on heuristic KV Cache policies, which ignore differences in token importance in long-term video generation. This leads to the loss of critical spatiotemporal information and the accumulation of redundant, invalid cache, thereby degrading video generation quality and efficiency. To address this limitation, we first observe that token contributions to video generation are highly time-heterogeneous and accordingly propose a novel Past- and Future-Informed KV Cache Policy (PaFu-KV). Specifically, PaFu-KV introduces a lightweight Salience Estimation Head distilled from a bidirectional teacher to estimate salience scores, allowing the KV cache to retain informative tokens while discarding less relevant ones. This policy yields a better quality-efficiency trade-off by shrinking KV cache capacity and reducing memory footprint at inference time. Extensive experiments on benchmarks demonstrate that our method preserves high-fidelity video generation quality while enables accelerated inference, thereby enabling more efficient long-horizon video generation. Our code will be released upon paper acceptance.