mlsanityData to 14 Oct 2026

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

arXiv 2206.10789 ↗First posted 22 Jun 2022cs.CV cs.LGcited by 1,295

Not on the map. Why?

Cited by 499 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, akin to machine translation, with sequences of image tokens as the target outputs rather than text tokens in another language. This strategy can naturally tap into the rich body of prior work on large language models, which have seen continued advances in capabilities and performance through scaling data and model sizes. Our approach is simple: First, Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. Second, we achieve consistent quality improvements by scaling the encoder-decoder Transformer model up to 20B parameters, with a new state-of-the-art zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO. Our detailed analysis on Localized Narratives as well as PartiPrompts (P2), a new holistic benchmark of over 1600 English prompts, demonstrate the effectiveness of Parti across a wide variety of categories and difficulty aspects. We also explore and highlight limitations of our models in order to define and exemplify key areas of focus for further improvements. See https://parti.research.google/ for high-resolution images.

Key facts. Cited by 1,295 papers in the map (data through 14 Oct 2026). Builds on 77 of its 115 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

77 of 115 references.

References · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  13. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  14. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  15. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  16. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  17. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  18. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  19. 2018Datasheets for Datasetscited by 1,386
  20. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  21. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  22. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  23. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  24. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  25. 2018Model Cards for Model Reportingcited by 1,080
  26. 2021Scaling Vision Transformerscited by 999
  27. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  28. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  29. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  30. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  31. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  32. 2016Generative Adversarial Text to Image Synthesiscited by 735
  33. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  34. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  35. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  36. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  37. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  38. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 101; 2023: 370; 2024: 372; 2025: 296; 2026: 156.

1,295 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
38 more
  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  3. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  4. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  5. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  6. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  7. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  8. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  12. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  13. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  14. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  15. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  16. 2023Adversarial Diffusion Distillationcited by 759
  17. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  18. 2022On Distillation of Guided Diffusion Modelscited by 725
  19. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  20. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  21. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  22. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 684
  23. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  24. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  25. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  26. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  27. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  28. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  29. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  30. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  31. 2023Zero-shot Image-to-Image Translationcited by 505
  32. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  33. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  34. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  35. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  36. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  37. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  38. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475

50 most cited of 1,295 All 1,295 (JSON)