# Autoregressive Video Generation without Vector Quantization

arXiv: https://arxiv.org/abs/2412.14169 · HTML: https://mlsanity.com/paper/2412.14169/

## Key facts

- Cited by 186 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- First posted on arXiv 18 Dec 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Built on by (most cited first, 30 of 186)

- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [LongLive: Real-time Interactive Long Video Generation](https://mlsanity.com/paper/2509.22622/) (2025)
- [OmniGen2: Towards Instruction-Aligned Multimodal Generation](https://mlsanity.com/paper/2506.18871/) (2025)
- [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025)
- [Self-Forcing++: Towards Minute-Scale High-Quality Video Generation](https://mlsanity.com/paper/2510.02283/) (2025)
- [Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation](https://mlsanity.com/paper/2602.02214/) (2026)
- [Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval](https://mlsanity.com/paper/2506.03141/) (2025)
- [Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation](https://mlsanity.com/paper/2505.05472/) (2025)
- [Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation](https://mlsanity.com/paper/2508.09987/) (2025)
- [Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rollout](https://mlsanity.com/paper/2511.20649/) (2025)
- [Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation](https://mlsanity.com/paper/2512.04678/) (2025)
- [MotionStream: Real-Time Video Generation with Interactive Motion Controls](https://mlsanity.com/paper/2511.01266/) (2025)
- [Helios: Real Real-Time Long Video Generation Model](https://mlsanity.com/paper/2603.04379/) (2026)
- [Dual Data Alignment Makes AI-Generated Image Detector Easier Generalizable](https://mlsanity.com/paper/2505.14359/) (2025)
- [Context Forcing: Consistent Autoregressive Video Generation with Long Context](https://mlsanity.com/paper/2602.06028/) (2026)
- [GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation](https://mlsanity.com/paper/2504.02782/) (2025)
- [UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions](https://mlsanity.com/paper/2506.13691/) (2025)
- [Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation](https://mlsanity.com/paper/2605.15141/) (2026)
- [End-to-End Training for Autoregressive Video Diffusion via Self-Resampling](https://mlsanity.com/paper/2512.15702/) (2025)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems](https://mlsanity.com/paper/2504.02918/) (2025)
- [InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation](https://mlsanity.com/paper/2511.04675/) (2025)
- [MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives](https://mlsanity.com/paper/2512.14699/) (2025)
- [VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory](https://mlsanity.com/paper/2512.04519/) (2025)
- [Video-T1: Test-Time Scaling for Video Generation](https://mlsanity.com/paper/2503.18942/) (2025)
- [LoL: Longer than Longer, Scaling Video Generation to Hour](https://mlsanity.com/paper/2601.16914/) (2026)
- [Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion](https://mlsanity.com/paper/2602.07775/) (2026)
- [Energy-Based Transformers are Scalable Learners and Thinkers](https://mlsanity.com/paper/2507.02092/) (2025)
- [SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation](https://mlsanity.com/paper/2507.09862/) (2025)

## Abstract (arXiv)

This paper presents a novel approach that enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction and spatial set-by-set prediction. Unlike raster-scan prediction in prior autoregressive models or joint distribution modeling of fixed-length tokens in diffusion models, our approach maintains the causal property of GPT-style models for flexible in-context capabilities, while leveraging bidirectional modeling within individual frames for efficiency. With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity, i.e., 0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models in text-to-image generation tasks, with a significantly lower training cost. Additionally, NOVA generalizes well across extended video durations and enables diverse zero-shot applications in one unified model. Code and models are publicly available at https://github.com/baaivision/NOVA.