# VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling

arXiv: https://arxiv.org/abs/2408.01181 · HTML: https://mlsanity.com/paper/2408.01181/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 55 references that are in the map.
- First posted on arXiv 2 Aug 2024.

## Builds on (most cited first, 30 of 49)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)

## Built on by (most cited first, 30 of 42)

- [HART: Efficient Visual Generation with Hybrid Autoregressive Transformer](https://mlsanity.com/paper/2410.10812/) (2024)
- [Randomized Autoregressive Visual Generation](https://mlsanity.com/paper/2411.00776/) (2024)
- [VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model](https://mlsanity.com/paper/2501.12327/) (2025)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient](https://mlsanity.com/paper/2411.17787/) (2024)
- [Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis](https://mlsanity.com/paper/2412.01819/) (2024)
- [R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation](https://mlsanity.com/paper/2505.23493/) (2025)
- [CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction](https://mlsanity.com/paper/2412.06782/) (2024)
- [EmotiCrafter: Text-to-Emotional-Image Generation based on Valence-Arousal Model](https://mlsanity.com/paper/2501.05710/) (2025)
- [FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction](https://mlsanity.com/paper/2502.20313/) (2025)
- [Customize Your Visual Autoregressive Recipe with Set Autoregressive Modeling](https://mlsanity.com/paper/2410.10511/) (2024)
- [Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression](https://mlsanity.com/paper/2505.19602/) (2025)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation](https://mlsanity.com/paper/2506.04421/) (2025)
- [Visual Generation Without Guidance](https://mlsanity.com/paper/2501.15420/) (2025)
- [ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation](https://mlsanity.com/paper/2511.11483/) (2025)
- [X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models](https://mlsanity.com/paper/2412.01824/) (2024)
- [Privacy Attacks on Image AutoRegressive Models](https://mlsanity.com/paper/2502.02514/) (2025)
- [DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizer](https://mlsanity.com/paper/2507.04947/) (2025)
- [MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization](https://mlsanity.com/paper/2504.00999/) (2025)
- [SpectralAR: Spectral Autoregressive Visual Generation](https://mlsanity.com/paper/2506.10962/) (2025)
- [Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspective](https://mlsanity.com/paper/2410.22217/) (2024)
- [MoSa: Motion Generation with Scalable Autoregressive Modeling](https://mlsanity.com/paper/2511.01200/) (2025)
- [Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression](https://mlsanity.com/paper/2506.09482/) (2025)
- [SATURN: Autoregressive Image Generation Guided by Scene Graphs](https://mlsanity.com/paper/2508.14502/) (2025)
- [Conditioned Activation Transport for T2I Safety Steering](https://mlsanity.com/paper/2603.03163/) (2026)
- [Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation](https://mlsanity.com/paper/2510.21003/) (2025)
- [AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers](https://mlsanity.com/paper/2511.16047/) (2025)
- [Markovian Scale Prediction: A New Era of Visual Autoregressive Generation](https://mlsanity.com/paper/2511.23334/) (2025)
- [Next-Scale Autoregressive Models are Zero-Shot Single-Image Object View Synthesizers](https://mlsanity.com/paper/2503.13588/) (2025)

## Abstract (arXiv)

VAR is a new generation paradigm that employs 'next-scale prediction' as opposed to 'next-token prediction'. This innovative transformation enables auto-regressive (AR) transformers to rapidly learn visual distributions and achieve robust generalization. However, the original VAR model is constrained to class-conditioned synthesis, relying solely on textual captions for guidance. In this paper, we introduce VAR-CLIP, a novel text-to-image model that integrates Visual Auto-Regressive techniques with the capabilities of CLIP. The VAR-CLIP framework encodes captions into text embeddings, which are then utilized as textual conditions for image generation. To facilitate training on extensive datasets, such as ImageNet, we have constructed a substantial image-text dataset leveraging BLIP2. Furthermore, we delve into the significance of word positioning within CLIP for the purpose of caption guidance. Extensive experiments confirm VAR-CLIP's proficiency in generating fantasy images with high fidelity, textual congruence, and aesthetic excellence. Our project page are https://github.com/daixiangzi/VAR-CLIP