mlsanityData to 7 Oct 2026

FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning

arXiv 2503.23367 ↗First posted 30 Mar 2025cs.CVcited by 34

On the map: Efficient LLM inference and serving → Fast autoregressive visual generation

Abstract (arXiv)

Visual Autoregressive (VAR) modeling has gained popularity for its shift towards next-scale prediction. However, existing VAR paradigms process the entire token map at each scale step, leading to the complexity and runtime scaling dramatically with image resolution. To address this challenge, we propose FastVAR, a post-training acceleration method for efficient resolution scaling with VARs. Our key finding is that the majority of latency arises from the large-scale step where most tokens have already converged. Leveraging this observation, we develop the cached token pruning strategy that only forwards pivotal tokens for scale-specific modeling while using cached tokens from previous scale steps to restore the pruned slots. This significantly reduces the number of forwarded tokens and improves the efficiency at larger resolutions. Experiments show the proposed FastVAR can further speedup FlashAttention-accelerated VAR by 2.7× with negligible performance drop of <1%. We further extend FastVAR to zero-shot generation of higher resolution images. In particular, FastVAR can generate one 2K image with 15GB memory footprints in 1.5s on a single NVIDIA 3090 GPU. Code is available at https://github.com/csguoh/FastVAR.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Fast autoregressive visual generation, one of 12 directions in Efficient LLM inference and serving. Builds on 60 of its 64 references that are in the map. First posted on arXiv 30 Mar 2025.

Builds on

60 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding2 Oct

Other references · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  9. 2022Token Merging: Your ViT But Fastercited by 995
  10. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  11. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  12. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  13. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  14. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  15. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  16. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  17. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  18. 2022On Distillation of Guided Diffusion Modelscited by 721
  19. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  20. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  21. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  22. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  23. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  24. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  25. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  26. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  27. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  28. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  29. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  30. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  31. 2022Post-training Quantization on Diffusion Modelscited by 262
  32. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  33. 2023Token Merging for Fast Stable Diffusioncited by 220
  34. 2023Structural Pruning for Diffusion Modelscited by 193
  35. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  36. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  37. 2024Accelerating Diffusion Transformers with Token-wise Feature Cachingcited by 143
  38. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 138

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 20.

34 papers.

Sort:

By area:

  1. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  2. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  3. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  4. 2025RestoreVAR: Visual Autoregressive Generation for All-in-One Image Restorationcited by 15
  5. 2025SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skippingcited by 11
  6. 2025MVAR: Visual Autoregressive Modeling with Scale and Spatial Markovian Conditioningcited by 8
  7. 2025Progressive Supernet Training for Efficient Visual Autoregressive Modelingcited by 5
  8. 2026ToProVAR: Efficient Visual Autoregressive Modeling via Tri-Dimensional Entropy-Aware Semantic Analysis and Sparsity Optimizationcited by 5
  9. 2026PreciseCache: Precise Feature Caching for Efficient and High-fidelity Video Generationcited by 5
  10. 2025SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world modelscited by 4
  11. 2026SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Accelerationcited by 4
  12. 2026RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Spacecited by 4
22 more
  1. 2026Multi-Scale Local Speculative Decoding for Image Generationcited by 3
  2. 2025Markovian Scale Prediction: A New Era of Visual Autoregressive Generationcited by 2
  3. 2025Autoregressive Image Generation Needs Only a Few Lines of Cached Tokenscited by 2
  4. 2025FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Modelscited by 2
  5. 2026FastSTAR: Spatiotemporal Token Pruning for Efficient Autoregressive Video Synthesiscited by 2
  6. 2025ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generationcited by 1
  7. 2026Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysiscited by 1
  8. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
  9. 2025Diversity Has Always Been There in Your Visual Autoregressive Modelscited by 0
  10. 2025DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Modelscited by 0
  11. 2026StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Modelscited by 0
  12. 2026Depth Adaptive Efficient Visual Autoregressive Modelingcited by 0
  13. 2026Visual Implicit Autoregressive Modelingcited by 0
  14. 2026ExtraVAR: Stage-Aware RoPE Remapping for Resolution Extrapolation in Visual Autoregressive Modelscited by 0
  15. 2026HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modelingcited by 0
  16. 2026Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generationcited by 0
  17. 2026MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observationscited by 0
  18. 2026MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Expertscited by 0
  19. 2026SynVAR: Synergizing Spatial and Semantic Alignment in Visual Autoregressive Modelcited by 0
  20. 2026SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesiscited by 0
  21. 2026Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modelingcited by 0
  22. 2026Edit-VAR: Taming Visual Autoregressive Model for Precise Video Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Frequency-Aware Autoregressive Modeling for Efficient High-Resolution Image Synthesis
  2. 2025FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction
  3. 2024M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation
  4. 2025ScalingAR: Scaling Confidence for Autoregressive Image Generation
  5. 2025Scale-Wise VAR is Secretly Discrete Diffusion
  6. 2025AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers
  7. 2025Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation
  8. 2026VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction