# Visual Autoregressive Modeling for Instruction-Guided Image Editing

arXiv: https://arxiv.org/abs/2508.15772 · HTML: https://mlsanity.com/paper/2508.15772/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 50 of its 55 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea](https://mlsanity.com/paper/2411.15738/) (2024)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [ImgEdit: A Unified Image Editing Dataset and Benchmark](https://mlsanity.com/paper/2505.20275/) (2025)
- [In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer](https://mlsanity.com/paper/2504.20690/) (2025)
- [Guiding Instruction-based Image Editing via Multimodal Large Language Models](https://mlsanity.com/paper/2309.17102/) (2023)
- [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://mlsanity.com/paper/2310.01506/) (2023)
- [OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision](https://mlsanity.com/paper/2411.07199/) (2024)

## Built on by (most cited first, 22 of 22)

- [HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer](https://mlsanity.com/paper/2605.11061/) (2026)
- [ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation](https://mlsanity.com/paper/2511.01163/) (2025)
- [EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editing](https://mlsanity.com/paper/2512.11715/) (2025)
- [LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation](https://mlsanity.com/paper/2604.11789/) (2026)
- [Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling](https://mlsanity.com/paper/2604.28185/) (2026)
- [IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection](https://mlsanity.com/paper/2603.29602/) (2026)
- [Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking](https://mlsanity.com/paper/2509.12046/) (2025)
- [MiLDEdit: Reasoning-Based Multi-Layer Design Document Editing](https://mlsanity.com/paper/2601.04589/) (2026)
- [Masked Generative Transformer Is What You Need for Image Editing](https://mlsanity.com/paper/2605.10859/) (2026)
- [Visual Generation Tuning](https://mlsanity.com/paper/2511.23469/) (2025)
- [Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams](https://mlsanity.com/paper/2608.12262/) (2026)
- [DreamVAR: Taming Reinforced Visual Autoregressive Model for High-Fidelity Subject-Driven Image Generation](https://mlsanity.com/paper/2601.22507/) (2026)
- [Rethinking Structure Preservation in Text-Guided Image Editing with Visual Autoregressive Models](https://mlsanity.com/paper/2603.28367/) (2026)
- [UniEditBench: A Unified and Cost-Effective Benchmark for Image and Video Editing via Distilled MLLMs](https://mlsanity.com/paper/2604.15871/) (2026)
- [Edit Where You Mean: Region-Aware Adapter Injection for Mask-Free Local Image Editing](https://mlsanity.com/paper/2604.23763/) (2026)
- [CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences](https://mlsanity.com/paper/2606.00931/) (2026)
- [Edit the Bits, Diff the Codes: Bitwise Residual Editing for Visual Autoregressive Models](https://mlsanity.com/paper/2606.13558/) (2026)
- [Next-Dense-Stride Prediction for Multimodal Autoregressive Visual Modeling](https://mlsanity.com/paper/2607.09892/) (2026)
- [ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning](https://mlsanity.com/paper/2607.12433/) (2026)
- [UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling](https://mlsanity.com/paper/2607.24157/) (2026)
- [Attribute Token Arithmetic: Disentangled and Continuous Semantic Control for Visual Autoregressive Models](https://mlsanity.com/paper/2608.28082/) (2026)
- [Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modeling](https://mlsanity.com/paper/2609.11804/) (2026)

## Abstract (arXiv)

Recent advances in diffusion models have brought remarkable visual fidelity to instruction-guided image editing. However, their global denoising process inherently entangles the edited region with the entire image context, leading to unintended spurious modifications and compromised adherence to editing instructions. In contrast, autoregressive models offer a distinct paradigm by formulating image synthesis as a sequential process over discrete visual tokens. Their causal and compositional mechanism naturally circumvents the adherence challenges of diffusion-based methods. In this paper, we present VAREdit, a visual autoregressive (VAR) framework that reframes image editing as a next-scale prediction problem. Conditioned on source image features and text instructions, VAREdit generates multi-scale target features to achieve precise edits. A core challenge in this paradigm is how to effectively condition the source image tokens. We observe that finest-scale source features cannot effectively guide the prediction of coarser target features. To bridge this gap, we introduce a Scale-Aligned Reference (SAR) module, which injects scale-matched conditioning information into the first self-attention layer. VAREdit demonstrates significant advancements in both editing adherence and efficiency. On EMU-Edit and PIE-Bench benchmarks, VAREdit outperforms leading diffusion-based methods by a substantial margin in terms of both CLIP and GPT scores. Moreover, VAREdit completes a 512×512 editing in 1.2 seconds, making it 2.2× faster than the similarly sized UltraEdit. Code is available at: https://github.com/HiDream-ai/VAREdit.