# LazyMAR: Accelerating Masked Autoregressive Models via Feature Caching

arXiv: https://arxiv.org/abs/2503.12450 · HTML: https://mlsanity.com/paper/2503.12450/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Fast autoregressive visual generation, one of 12 directions in Efficient LLM inference and serving.
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Placed in

- Efficient LLM inference and serving / [Fast autoregressive visual generation](https://mlsanity.com/inference/fast-autoregressive-visual-generation/)

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [An Image is Worth 32 Tokens for Reconstruction and Generation](https://mlsanity.com/paper/2406.07550/) (2024)
- [MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis](https://mlsanity.com/paper/2211.09117/) (2022)
- [Accelerating Diffusion Transformers with Token-wise Feature Caching](https://mlsanity.com/paper/2410.05317/) (2024)
- [DiT4Edit: Diffusion Transformer for Image Editing](https://mlsanity.com/paper/2411.03286/) (2024)
- [Deep AutoRegressive Networks](https://mlsanity.com/paper/1310.8499/) (2013)
- [Parallelized Autoregressive Visual Generation](https://mlsanity.com/paper/2412.15119/) (2024)
- [LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding](https://mlsanity.com/paper/2410.03355/) (2024)
- [NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time](https://mlsanity.com/paper/2408.03675/) (2024)
- [Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation](https://mlsanity.com/paper/2409.01055/) (2024)

## Built on by (most cited first, 11 of 11)

- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformers](https://mlsanity.com/paper/2511.20123/) (2025)
- [DiSA: Diffusion Step Annealing in Autoregressive Image Generation](https://mlsanity.com/paper/2505.20297/) (2025)
- [Generation then Reconstruction: Accelerating Masked Autoregressive Models via Two-Stage Sampling](https://mlsanity.com/paper/2510.17171/) (2025)
- [Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens](https://mlsanity.com/paper/2512.04857/) (2025)
- [MARch'e: Fast Masked Autoregressive Image Generation with Cache-Aware Attention](https://mlsanity.com/paper/2506.12035/) (2025)
- [AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs](https://mlsanity.com/paper/2509.11155/) (2025)
- [LINA: Linear Autoregressive Image Generative Models with Continuous Tokens](https://mlsanity.com/paper/2601.22630/) (2026)
- [Drift-AR: Single-Step Visual Autoregressive Generation via Anti-Symmetric Drifting](https://mlsanity.com/paper/2603.28049/) (2026)
- [XYZFlow:Scaling Multi dimensional Shortcut Flows for Efficient Generative Modeling](https://mlsanity.com/paper/2608.12276/) (2026)
- [FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models](https://mlsanity.com/paper/2609.36202/) (2026)

## Abstract (arXiv)

Masked Autoregressive (MAR) models have emerged as a promising approach in image generation, expected to surpass traditional autoregressive models in computational efficiency by leveraging the capability of parallel decoding. However, their dependence on bidirectional self-attention inherently conflicts with conventional KV caching mechanisms, creating unexpected computational bottlenecks that undermine their expected efficiency. To address this problem, this paper studies the caching mechanism for MAR by leveraging two types of redundancy: Token Redundancy indicates that a large portion of tokens have very similar representations in the adjacent decoding steps, which allows us to first cache them in previous steps and then reuse them in the later steps. Condition Redundancy indicates that the difference between conditional and unconditional output in classifier-free guidance exhibits very similar values in adjacent steps. Based on these two redundancies, we propose LazyMAR, which introduces two caching mechanisms to handle them one by one. LazyMAR is training-free and plug-and-play for all MAR models. Experimental results demonstrate that our method achieves 2.83 times acceleration with almost no drop in generation quality. Our codes will be released in https://github.com/feihongyan1/LazyMAR.