mlsanityData to 7 Oct 2026

VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling

arXiv 2408.01181 ↗First posted 2 Aug 2024cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

VAR is a new generation paradigm that employs 'next-scale prediction' as opposed to 'next-token prediction'. This innovative transformation enables auto-regressive (AR) transformers to rapidly learn visual distributions and achieve robust generalization. However, the original VAR model is constrained to class-conditioned synthesis, relying solely on textual captions for guidance. In this paper, we introduce VAR-CLIP, a novel text-to-image model that integrates Visual Auto-Regressive techniques with the capabilities of CLIP. The VAR-CLIP framework encodes captions into text embeddings, which are then utilized as textual conditions for image generation. To facilitate training on extensive datasets, such as ImageNet, we have constructed a substantial image-text dataset leveraging BLIP2. Furthermore, we delve into the significance of word positioning within CLIP for the purpose of caption guidance. Extensive experiments confirm VAR-CLIP's proficiency in generating fantasy images with high fidelity, textual congruence, and aesthetic excellence. Our project page are https://github.com/daixiangzi/VAR-CLIP

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 49 of its 55 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

49 of 55 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  10. 2020Denoising Diffusion Implicit Modelscited by 9,655
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
37 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  13. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  14. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  15. 2022Video Diffusion Modelscited by 2,280
  16. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  17. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  18. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  19. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  20. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  21. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  22. 2016Pixel Recurrent Neural Networkscited by 814
  23. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  24. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  25. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  26. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  27. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  28. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  29. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  30. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  31. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  32. 2023Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Followingcited by 77
  33. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  34. 2023Controllable Text-to-Image Generation with GPT-4cited by 58
  35. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  36. 2024STAR: Scale-wise Text-conditioned AutoRegressive image generationcited by 28
  37. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 27; 2026: 6.

42 papers.

Sort:

By area:

  1. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  2. 2024Randomized Autoregressive Visual Generationcited by 125
  3. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  4. 2024Autoregressive Models in Vision: A Surveycited by 39
  5. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesiscited by 33
  6. 2024Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficientcited by 32
  7. 2025R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generationcited by 31
  8. 2024CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Predictioncited by 26
  9. 2025EmotiCrafter: Text-to-Emotional-Image Generation based on Valence-Arousal Modelcited by 23
  10. 2025FlexVAR: Flexible Visual Autoregressive Modeling without Residual Predictioncited by 22
  11. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  12. 2024Customize Your Visual Autoregressive Recipe with Set Autoregressive Modelingcited by 21
30 more
  1. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  2. 2025HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generationcited by 19
  3. 2025Visual Generation Without Guidancecited by 17
  4. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Modelscited by 15
  5. 2025Privacy Attacks on Image AutoRegressive Modelscited by 15
  6. 2025ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generationcited by 14
  7. 2025DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizercited by 13
  8. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  9. 2025SpectralAR: Spectral Autoregressive Visual Generationcited by 11
  10. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  11. 2025MoSa: Motion Generation with Scalable Autoregressive Modelingcited by 5
  12. 2025Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregressioncited by 4
  13. 2025SATURN: Autoregressive Image Generation Guided by Scene Graphscited by 3
  14. 2026Conditioned Activation Transport for T2I Safety Steeringcited by 3
  15. 2025Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillationcited by 2
  16. 2025AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformerscited by 2
  17. 2025Markovian Scale Prediction: A New Era of Visual Autoregressive Generationcited by 2
  18. 2025Next-Scale Autoregressive Models are Zero-Shot Single-Image Object View Synthesizerscited by 1
  19. 2025CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translationcited by 1
  20. 2026Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysiscited by 1
  21. 2026SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generationcited by 1
  22. 2025D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokenscited by 0
  23. 2025Efficient Multi-Instance Generation with Janus-Pro-Dirven Prompt Parsingcited by 0
  24. 2025Capturing Conditional Dependence via Auto-regressive Diffusion Modelscited by 0
  25. 2025Efficient Conditional Generation on Scale-based Visual Autoregressive Modelscited by 0
  26. 2025DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Modelscited by 0
  27. 2025Rethinking Training Dynamics in Scale-wise Autoregressive Generationcited by 0
  28. 2026SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editingcited by 0
  29. 2026Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoningcited by 0
  30. 2026Photorealistic Novel View Synthesis of Human Faces using Next-Scale Transformerscited by 0