mlsanityData to 14 Oct 2026

ControlVAR: Exploring Controllable Visual Autoregressive Modeling

arXiv 2406.09750 ↗First posted 14 Jun 2024cs.CV cs.AIcited by 58

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Conditional visual generation has witnessed remarkable progress with the advent of diffusion models (DMs), especially in tasks like control-to-image generation. However, challenges such as expensive computational cost, high inference latency, and difficulties of integration with large language models (LLMs) have necessitated exploring alternatives to DMs. This paper introduces ControlVAR, a novel framework that explores pixel-level controls in visual autoregressive (VAR) modeling for flexible and efficient conditional generation. In contrast to traditional conditional models that learn the conditional distribution, ControlVAR jointly models the distribution of image and pixel-level conditions during training and imposes conditional controls during testing. To enhance the joint modeling, we adopt the next-scale AR prediction paradigm and unify control and image representations. A teacher-forcing guidance strategy is proposed to further facilitate controllable generation with joint modeling. Extensive experiments demonstrate the superior efficacy and flexibility of ControlVAR across various conditional generation tasks against popular conditional DMs, eg, ControlNet and T2I-Adaptor. Code: urlhttps://github.com/lxa9867/ControlVAR.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 85 of its 98 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

85 of 98 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction3 Apr
  2. 2024CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion8 Mar
  3. 2024DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization15 Feb
  4. 2023X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model4 Dec
  5. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models25 May
  6. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild18 May
  7. 2023Region-Aware Diffusion for Zero-shot Text-driven Image Editing23 Feb
  8. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  9. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  12. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
6 more
  1. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  3. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Model15 Nov
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2023Segment Anythingcited by 8,234
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Scalable Diffusion Models with Transformerscited by 6,054
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,847
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2016Improved Techniques for Training GANscited by 2,970
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  9. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  10. 2023Consistency Modelscited by 1,822
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  12. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  13. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  14. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  15. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  16. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  17. 2018Self-Attention Generative Adversarial Networkscited by 809
  18. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  19. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  20. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  21. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  22. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  23. 2021Score-based Generative Modeling in Latent Spacecited by 636
  24. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  25. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  26. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  27. 2023Improved Techniques for Training Consistency Modelscited by 439
  28. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  29. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  30. 2019SOLO: Segmenting Objects by Locationscited by 289
  31. 2019DIODE: A Dense Indoor and Outdoor DEpth Datasetcited by 240
  32. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  33. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  34. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 179
  35. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  36. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  37. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  38. 2023What's In My Big Data?cited by 129

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 32; 2026: 13.

58 papers.

Sort:

By area:

  1. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
  2. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 132
  3. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 94
  4. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  5. 2024ControlAR: Controllable Image Generation with Autoregressive Modelscited by 48
  6. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  7. 2024VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modelingcited by 42
  8. 2025EditAR: Unified Conditional Generation with Autoregressive Modelscited by 40
  9. 2024Autoregressive Models in Vision: A Surveycited by 39
  10. 2024Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficientcited by 33
  11. 2024CAR: Controllable Autoregressive Modeling for Visual Generationcited by 30
  12. 2025Neighboring Autoregressive Modeling for Efficient Visual Generationcited by 29
38 more
  1. 2024STAR: Scale-wise Text-conditioned AutoRegressive image generationcited by 28
  2. 2024CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Predictioncited by 27
  3. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  4. 2025Autoregressive Image Generation with Randomized Parallel Decodingcited by 20
  5. 2025HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generationcited by 20
  6. 2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
  7. 2025Frequency-Aware Autoregressive Modeling for Efficient High-Resolution Image Synthesiscited by 15
  8. 2025NFIG: Multi-Scale Autoregressive Image Generation via Frequency Orderingcited by 14
  9. 2025Fine-Tuning Visual Autoregressive Models for Subject-Driven Generationcited by 13
  10. 2025DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizercited by 13
  11. 2025RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligencecited by 13
  12. 2024XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generationcited by 12
  13. 2025Image Tokenizer Needs Post-Trainingcited by 12
  14. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  15. 2025Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesiscited by 10
  16. 2025PRISM: A Unified Framework for Photorealistic Reconstruction and Intrinsic Scene Modelingcited by 9
  17. 2025ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoningcited by 8
  18. 2025Semantic Context Matters: Improving Conditioning for Autoregressive Modelscited by 7
  19. 2025ICAS: Detecting Training Data from Autoregressive Image Generative Modelscited by 6
  20. 2025MoSa: Motion Generation with Scalable Autoregressive Modelingcited by 5
  21. 2025Progressive Supernet Training for Efficient Visual Autoregressive Modelingcited by 5
  22. 2024PanoLlama: Generating Endless and Coherent Panoramas with Next-Token-Prediction LLMscited by 4
  23. 2025Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Maskingcited by 3
  24. 2025EchoGen: Generating Visual Echoes in Any Scene via Feed-Forward Subject-Driven Auto-Regressive Modelcited by 3
  25. 2025EarthMapper: Visual Autoregressive Models for Controllable Bidirectional Satellite-Map Translationcited by 2
  26. 2025PacTure: Efficient PBR Texture Generation on Packed Views with Visual Autoregressive Modelscited by 2
  27. 2025ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attentioncited by 2
  28. 2025Seg-VAR: Image Segmentation with Visual Autoregressive Modelingcited by 2
  29. 2025Markovian Scale Prediction: A New Era of Visual Autoregressive Generationcited by 2
  30. 2025CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translationcited by 1
  31. 2026RemoteVAR: Autoregressive Visual Modeling for Remote Sensing Change Detectioncited by 1
  32. 2026DreamVAR: Taming Reinforced Visual Autoregressive Model for High-Fidelity Subject-Driven Image Generationcited by 1
  33. 2026Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysiscited by 1
  34. 2026VARestorer: One-Step VAR Distillation for Real-World Image Super-Resolutioncited by 1
  35. 2026Design Your Ad: Personalized Advertising Image and Text Generation with Unified Autoregressive Modelscited by 1
  36. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
  37. 2026Shift-and-Sum Quantization for Visual Autoregressive Modelscited by 1
  38. 2026VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizerscited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback
  2. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  3. 2025FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction
  4. 2024Self-control: A Better Conditional Mechanism for Masked Autoregressive Model
  5. 2026VISTA: Test-Time Compositional Alignment for Visual Autoregressive Generation
  6. 2025Scale-Wise VAR is Secretly Discrete Diffusion
  7. 2024A Simple Approach to Unifying Diffusion-based Conditional Generation
  8. 2025Diversity Has Always Been There in Your Visual Autoregressive Models