# Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation

arXiv: https://arxiv.org/abs/2606.00310 · HTML: https://mlsanity.com/paper/2606.00310/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 42 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 30 of 36)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment](https://mlsanity.com/paper/2403.05135/) (2024)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024)
- [TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation](https://mlsanity.com/paper/2412.03069/) (2024)
- [Accelerating Diffusion Transformers with Token-wise Feature Caching](https://mlsanity.com/paper/2410.05317/) (2024)
- [Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining](https://mlsanity.com/paper/2408.02657/) (2024)
- [Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching](https://mlsanity.com/paper/2406.01733/) (2024)
- [HART: Efficient Visual Generation with Hybrid Autoregressive Transformer](https://mlsanity.com/paper/2410.10812/) (2024)
- [LMFusion: Adapting Pretrained Language Models for Multimodal Generation](https://mlsanity.com/paper/2412.15188/) (2024)
- [DiT4Edit: Diffusion Transformer for Image Editing](https://mlsanity.com/paper/2411.03286/) (2024)
- [SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL](https://mlsanity.com/paper/2504.11455/) (2025)
- [Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding](https://mlsanity.com/paper/2410.01699/) (2024)
- [Follow-Your-Creation: Empowering 4D Creation through Video Inpainting](https://mlsanity.com/paper/2506.04590/) (2025)
- [LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding](https://mlsanity.com/paper/2410.03355/) (2024)
- [Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models](https://mlsanity.com/paper/2405.05252/) (2024)
- [LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers](https://mlsanity.com/paper/2412.12444/) (2024)
- [Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis](https://mlsanity.com/paper/2411.07132/) (2024)

## Abstract (arXiv)

Visual Autoregressive (VAR) models deliver high-quality image generation but suffer from significant inference latency at high resolutions. Recent acceleration approaches most rely on heuristic measures with layer features to prune tokens. Such heuristics are sensitive to complex contextual semantics, leading to inaccurate identification of redundant computation and poor adaptability across prompts. We rethink redundancy in VAR from the perspective of its impact on pixel-space generation and introduce Latent Discrepancy. This unified metric quantifies a token's contribution by measuring the change in model states during generation. Our analysis shows that redundancy is more accurately identified when guided by image latent or pixel-space signals. We further observed that in classifier-free guidance (CFG), the convergence trend of the discrepancy between conditional and unconditional branches exhibits high dynamics with different prompts. Based on these findings, we propose LD-Pruning (Latent Discrepancy Pruning), a training-free framework that removes redundancy via latent discrepancy by integrating decoding-free region selection and adaptive unconditional-branch skipping. Extensive experiments show that LD-Pruning substantially reduces inference latency while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.