mlsanityData to 14 Oct 2026

Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models

arXiv 2504.12626 ↗First posted 17 Apr 2025cs.CVcited by 143

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation. FramePack compresses input frame contexts with frame-wise importance so that more frames can be encoded within a fixed context length, with more important frames having longer contexts. The frame importance can be measured using time proximity, feature similarity, or hybrid metrics. The packing method allows for inference with thousands of frames and training with relatively large batch sizes. We also present drift prevention methods to address observation bias (error accumulation), including early-established endpoints, adjusted sampling orders, and discrete history representation. Ablation studies validate the effectiveness of the anti-drifting methods in both single-directional video streaming and bi-directional video generation. Finally, we show that existing video diffusion models can be finetuned with FramePack, and analyze the differences between different packing schedules.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 61 of its 74 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

61 of 74 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Framework12 Mar
  2. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity3 Feb
  3. 2024LTX-Video: Realtime Video Latent Diffusion30 Dec
  4. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation24 Dec
  5. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model28 Nov
  6. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality25 Oct
  7. 2024FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention29 Jul
  8. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence23 Jul
  9. 2024Video-Infinity: Distributed Long Video Generation24 Jun
  10. 2024ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Models16 Jun
  11. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  12. 2024VideoStudio: Generating Consistent-Content and Multi-Scene Videos2 Jan
7 more
  1. 2023MEVG: Multi-event Video Generation with Text-to-Video Models7 Dec
  2. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Models30 Nov
  3. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling23 Oct
  4. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  5. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  6. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  7. 2022Flexible Diffusion Modeling of Long Videos23 May

Other references · 42

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2023Consistency Modelscited by 1,822
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  10. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,081
30 more
  1. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  2. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  3. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 684
  4. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  5. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  6. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  7. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  8. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  9. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  10. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  11. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  12. 20243D Reconstruction with Spatial Memorycited by 277
  13. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  14. 2020Real-Time Intermediate Flow Estimation for Video Frame Interpolationcited by 213
  15. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 202
  16. 2025History-Guided Video Diffusioncited by 171
  17. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  18. 2019RetinaFace: Single-stage Dense Face Localisation in the Wildcited by 165
  19. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  20. 2023WonderJourney: Going from Anywhere to Everywherecited by 158
  21. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 123
  22. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 115
  23. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  24. 2025One-Minute Video Generation with Test-Time Trainingcited by 101
  25. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 40
  26. 2024DreamFactory: Pioneering Multi-Scene Long Video Generation with a Multi-Agent Frameworkcited by 35
  27. 2024StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaborationcited by 31
  28. 2024TALC: Time-Aligned Captions for Multi-Scene Text-to-Video Generationcited by 25
  29. 2024Synchronized Video Storytelling: Generating Video Narrations with Structured Storylinecited by 19
  30. 2025HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 44; 2026: 99.

143 papers.

Sort:

By area:

  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  2. 2025LongLive: Real-time Interactive Long Video Generationcited by 265
  3. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234
  4. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  5. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 170
  6. 2025Video World Models with Long-term Spatial Memorycited by 136
  7. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 106
  8. 2025Mixture of Contexts for Long Video Generationcited by 95
  9. 2025STream3R: Scalable Sequential 3D Reconstruction with Causal Transformercited by 82
  10. 2025Yume: An Interactive World Generation Modelcited by 81
  11. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 73
  12. 2025Large Video Planner Enables Generalizable Robot Controlcited by 70
38 more
  1. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  2. 2025Wan-S2V: Audio-Driven Cinematic Video Generationcited by 61
  3. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  4. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 44
  5. 2025DeepVerse: 4D Autoregressive Video Generation as a World Modelcited by 40
  6. 2026Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memorycited by 40
  7. 2026Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusioncited by 32
  8. 2025Pack and Force Your Memory: Long-form and Consistent Video Generationcited by 29
  9. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 26
  10. 2026AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillationcited by 26
  11. 2026VideoMemory: Toward Consistent Video Generation via Memory Integrationcited by 25
  12. 2026Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generationcited by 24
  13. 2026Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantizationcited by 23
  14. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 22
  15. 2025LongVie: Multimodal-Guided Controllable Ultra-Long Video Generationcited by 22
  16. 2025Macro-from-Micro Planning for High-Quality and Parallelized Autoregressive Long Video Generationcited by 21
  17. 2025Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesiscited by 21
  18. 2025LoViC: Efficient Long Video Generation with Context Compressioncited by 19
  19. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 18
  20. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 17
  21. 2025WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perceptioncited by 16
  22. 2026MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Enginescited by 16
  23. 2026Anchor Forcing: Anchor Memory and Tri-Region RoPE for Interactive Streaming Video Diffusioncited by 16
  24. 2026Pathwise Test-Time Correction for Autoregressive Long Video Generationcited by 15
  25. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 15
  26. 2026Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Modelscited by 15
  27. 2026Addressable Memory for Video World Modelscited by 15
  28. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
  29. 2026Alaya-EVOKE: From Linear-Scaling Supervision to Endless Worldcited by 14
  30. 2025Reinforcement Learning with Inverse Rewards for World Model Post-trainingcited by 13
  31. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 13
  32. 2026EchoWM: Open and Enterable Omnimodal World Modelscited by 13
  33. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 12
  34. 2026Mode Seeking meets Mean Seeking for Fast Long Video Generationcited by 12
  35. 2026FlowAct-R1: Towards Interactive Humanoid Video Generationcited by 11
  36. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generationcited by 10
  37. 2025PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesiscited by 10
  38. 2026CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narrativescited by 10

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Streaming Video Diffusion Model: Context, Execution, and Training
  2. 2026PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference
  3. 2024Frame by Familiar Frame: Understanding Replication in Video Diffusion Models
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  5. 2023Video Diffusion Models with Local-Global Context Guidance
  6. 2025Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimization
  7. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusion
  8. 2022Diffusion Probabilistic Modeling for Video Generation