mlsanityData to 7 Oct 2026

LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization

arXiv 2411.17178 ↗First posted 26 Nov 2024cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Visual Autoregressive (VAR) has emerged as a promising approach in image generation, offering competitive potential and performance comparable to diffusion-based models. However, current AR-based visual generation models require substantial computational resources, limiting their applicability on resource-constrained devices. To address this issue, we conducted analysis and identified significant redundancy in three dimensions of the VAR model: (1) the attention map, (2) the attention outputs when using classifier free guidance, and (3) the data precision. Correspondingly, we proposed efficient attention mechanism and low-bit quantization method to enhance the efficiency of VAR models while maintaining performance. With negligible performance lost (less than 0.056 FID increase), we could achieve 85.2% reduction in attention computation, 50% reduction in overall memory and 1.5x latency reduction. To ensure deployment feasibility, we developed efficient training-free compression techniques and analyze the deployment feasibility and efficiency gain of each technique.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 20 of its 21 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

20 of 21 references.

References · 20

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  10. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  11. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
8 more
  1. 2021A White Paper on Neural Network Quantizationcited by 453
  2. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  5. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  6. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  7. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  8. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Modelscited by 109

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  2. 2025TR-DQ: Time-Rotation Diffusion Quantizationcited by 18
  3. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  4. 2025SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skippingcited by 11
  5. 2026PTQ4ARVG: Post-Training Quantization for AutoRegressive Visual Generation Modelscited by 3
  6. 2025AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformerscited by 2
  7. 2025Markovian Scale Prediction: A New Era of Visual Autoregressive Generationcited by 2
  8. 2025FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Modelscited by 2
  9. 2025Dynamic Mixture-of-Experts for Visual Autoregressive Modelcited by 1
  10. 2025ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generationcited by 1
  11. 2026Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysiscited by 1
  12. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
4 more
  1. 2025FPQVAR: Floating Point Quantization for Visual Autoregressive Model with FPGA Hardware Co-designcited by 0
  2. 2026Depth Adaptive Efficient Visual Autoregressive Modelingcited by 0
  3. 2026HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modelingcited by 0
  4. 2026Shift-and-Sum Quantization for Visual Autoregressive Modelscited by 0