# Neighboring Autoregressive Modeling for Efficient Visual Generation

arXiv: https://arxiv.org/abs/2503.10696 · HTML: https://mlsanity.com/paper/2503.10696/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 65 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)

## Built on by (most cited first, 29 of 29)

- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspective](https://mlsanity.com/paper/2507.08801/) (2025)
- [Autoregressive Image Generation with Randomized Parallel Decoding](https://mlsanity.com/paper/2503.10568/) (2025)
- [SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model](https://mlsanity.com/paper/2505.22126/) (2025)
- [Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropy](https://mlsanity.com/paper/2510.09012/) (2025)
- [UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2506.17202/) (2025)
- [FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge](https://mlsanity.com/paper/2505.14709/) (2025)
- [Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation](https://mlsanity.com/paper/2507.01957/) (2025)
- [TC-Pad'e: Trajectory-Consistent Pad'e Approximation for Diffusion Acceleration](https://mlsanity.com/paper/2603.02943/) (2026)
- [SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperation](https://mlsanity.com/paper/2505.19151/) (2025)
- [Make It Efficient: Dynamic Sparse Attention for Autoregressive Image Generation](https://mlsanity.com/paper/2506.18226/) (2025)
- [ICAS: Detecting Training Data from Autoregressive Image Generative Models](https://mlsanity.com/paper/2507.05068/) (2025)
- [PTQ4ARVG: Post-Training Quantization for AutoRegressive Visual Generation Models](https://mlsanity.com/paper/2601.21238/) (2026)
- [BitDance: Scaling Autoregressive Generative Models with Binary Tokens](https://mlsanity.com/paper/2602.14041/) (2026)
- [LaVieID: Local Autoregressive Diffusion Transformers for Identity-Preserving Video Creation](https://mlsanity.com/paper/2508.07603/) (2025)
- [Exploiting Discriminative Codebook Prior for Autoregressive Image Generation](https://mlsanity.com/paper/2508.10719/) (2025)
- [Generation then Reconstruction: Accelerating Masked Autoregressive Models via Two-Stage Sampling](https://mlsanity.com/paper/2510.17171/) (2025)
- [Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?](https://mlsanity.com/paper/2511.08704/) (2025)
- [Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens](https://mlsanity.com/paper/2512.04857/) (2025)
- [SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation](https://mlsanity.com/paper/2606.20543/) (2026)
- [LSRS: Latent Scale Rejection Sampling for Visual Autoregressive Modeling](https://mlsanity.com/paper/2512.03796/) (2025)
- [Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysis](https://mlsanity.com/paper/2602.01345/) (2026)
- [Progressive Checkerboards for Autoregressive Multiscale Image Generation](https://mlsanity.com/paper/2602.03811/) (2026)
- [Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning](https://mlsanity.com/paper/2607.02963/) (2026)
- [Next-Dense-Stride Prediction for Multimodal Autoregressive Visual Modeling](https://mlsanity.com/paper/2607.09892/) (2026)
- [Context-weighted Discrete Flow Matching](https://mlsanity.com/paper/2607.21427/) (2026)
- [Efficient Training with Foresight: Multi-Token Auxiliary Supervision for Autoregressive Image Generation](https://mlsanity.com/paper/2608.25386/) (2026)
- [C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Space](https://mlsanity.com/paper/2608.28603/) (2026)
- [Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation](https://mlsanity.com/paper/2609.19702/) (2026)

## Abstract (arXiv)

Visual autoregressive models typically adhere to a raster-order ``next-token prediction" paradigm, which overlooks the spatial and temporal locality inherent in visual content. Specifically, visual tokens exhibit significantly stronger correlations with their spatially or temporally adjacent tokens compared to those that are distant. In this paper, we propose Neighboring Autoregressive Modeling (NAR), a novel paradigm that formulates autoregressive visual generation as a progressive outpainting procedure, following a near-to-far ``next-neighbor prediction" mechanism. Starting from an initial token, the remaining tokens are decoded in ascending order of their Manhattan distance from the initial token in the spatial-temporal space, progressively expanding the boundary of the decoded region. To enable parallel prediction of multiple adjacent tokens in the spatial-temporal space, we introduce a set of dimension-oriented decoding heads, each predicting the next token along a mutually orthogonal dimension. During inference, all tokens adjacent to the decoded tokens are processed in parallel, substantially reducing the model forward steps for generation. Experiments on ImageNet256× 256 and UCF101 demonstrate that NAR achieves 2.4× and 8.6× higher throughput respectively, while obtaining superior FID/FVD scores for both image and video generation tasks compared to the PAR-4X approach. When evaluating on text-to-image generation benchmark GenEval, NAR with 0.8B parameters outperforms Chameleon-7B while using merely 0.4 of the training data. Code is available at https://github.com/ThisisBillhe/NAR.