mlsanityData to 14 Oct 2026

Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens

arXiv 2410.13863 ↗First posted 17 Oct 2024cs.CV cs.LGcited by 147

Not on the map. Why?

Cited by 28 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scaling up autoregressive models in vision has not proven as beneficial as in large language models. In this work, we investigate this scaling problem in the context of text-to-image generation, focusing on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed raster order using BERT- or GPT-like transformer architectures. Our empirical results show that, while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends. Models based on continuous tokens achieve significantly better visual quality than those using discrete tokens. Furthermore, the generation order and attention mechanisms significantly affect the GenEval score: random-order models achieve notably better GenEval scores compared to raster-order models. Inspired by these findings, we train Fluid, a random-order autoregressive model on continuous tokens. Fluid 10.5B model achieves a new state-of-the-art zero-shot FID of 6.16 on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our findings and results will encourage future efforts to further bridge the scaling gap between vision and language models.

Key facts. Cited by 147 papers in the map (data through 14 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

49 of 58 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2024The Llama 3 Herd of Modelscited by 14,715
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
37 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  14. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  15. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  16. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  17. 2021Scaling Vision Transformerscited by 999
  18. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  19. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  20. 2016Pixel Recurrent Neural Networkscited by 816
  21. 2018Image Transformercited by 772
  22. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  23. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  24. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  25. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  26. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  27. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  28. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  29. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  30. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  31. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  32. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  33. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  34. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142
  35. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  36. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  37. 2013Deep AutoRegressive Networkscited by 61

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 93; 2026: 41.

147 papers.

Sort:

By area:

  1. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 304
  2. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  3. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 211
  4. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  5. 2024Randomized Autoregressive Visual Generationcited by 127
  6. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 123
  7. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  8. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 83
  9. 2025PixNerd: Pixel Neural Field Diffusioncited by 82
  10. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  11. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 69
  12. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 60
38 more
  1. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 58
  2. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 57
  3. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 57
  4. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 56
  5. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  6. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 55
  7. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  8. 2025Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokenscited by 50
  9. 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 45
  10. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  11. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  12. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
  13. 2024Autoregressive Models in Vision: A Surveycited by 39
  14. 2025Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modelingcited by 37
  15. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesiscited by 33
  16. 2025FlowTok: Flowing Seamlessly Across Text and Image Tokenscited by 33
  17. 2025STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesiscited by 32
  18. 2025Taming Teacher Forcing for Masked Autoregressive Video Generationcited by 30
  19. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  20. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  21. 2025Frequency Autoregressive Image Generation with Continuous Tokenscited by 27
  22. 2025Next-Embedding Prediction Makes Strong Vision Learnerscited by 23
  23. 2025Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspectivecited by 22
  24. 2025VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learningcited by 21
  25. 2025LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimizationcited by 20
  26. 2025Autoregressive Image Generation with Randomized Parallel Decodingcited by 20
  27. 2025Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Modelcited by 20
  28. 2025Latent Denoising Makes Good Tokenizerscited by 20
  29. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 18
  30. 2025Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Modelscited by 18
  31. 2025Visual Generation Without Guidancecited by 17
  32. 2025VideoMAR: Autoregressive Video Generatio with Continuous Tokenscited by 16
  33. 2025Watermarking Autoregressive Image Generationcited by 16
  34. 2025Hyperspherical Latents Improve Continuous-Token Autoregressive Generationcited by 16
  35. 2025Privacy Attacks on Image AutoRegressive Modelscited by 15
  36. 2025MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTscited by 14
  37. 2026PixelGen: Improving Pixel Diffusion with Perceptual Supervisioncited by 14
  38. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13

50 most cited of 147 All 147 (JSON)