mlsanityData to 7 Oct 2026

Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining

arXiv 2408.02657 ↗First posted 5 Aug 2024cs.CVcited by 148

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Lumina-mGPT, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions. By initializing from multimodal Generative PreTraining (mGPT), we demonstrate that decoder-only Autoregressive (AR) model can achieve image generation performance comparable to modern diffusion models with high efficiency through Flexible Progressive Supervised Fine-tuning (FP-SFT). Equipped with our proposed Unambiguous image Representation (UniRep), Lumina-mGPT can flexibly generate high-quality images of varying aspect ratios. Building on the strong image generation capabilities, we further explore Ominiponent Supervised Fine-tuning (Omni-SFT), an initial attempt to elevate Lumina-mGPT into a unified multi-modal generalist. The resulting model demonstrates versatile multimodal capabilities, including visual generation tasks like text-to-image/multiview generation and controllable generation, visual recognition tasks like segmentation and depth estimation, and vision-language tasks like multi-turn visual question answering, showing the rosy potential of the technical direction. Codes and checkpoints are available at https://github.com/Alpha-VLLM/Lumina-mGPT.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 76 of its 98 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

76 of 98 references.

References · 76

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  13. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  14. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  15. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  16. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  17. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  18. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  19. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  20. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  21. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  22. 2020GLU Variants Improve Transformercited by 1,724
  23. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  24. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  25. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  26. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  27. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  28. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  29. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  30. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  31. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  32. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  33. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  34. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  35. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  36. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  37. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  38. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 79; 2026: 42.

148 papers.

Sort:

By area:

  1. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  2. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  3. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  4. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  5. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 93
  6. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  7. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  8. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  9. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  10. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  11. 2024Parallelized Autoregressive Visual Generationcited by 58
  12. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
38 more
  1. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  2. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  3. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  4. 2024ControlAR: Controllable Image Generation with Autoregressive Modelscited by 48
  5. 2025Unified Multimodal Discrete Diffusioncited by 46
  6. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  7. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  8. 2025EditAR: Unified Conditional Generation with Autoregressive Modelscited by 40
  9. 2024Autoregressive Models in Vision: A Surveycited by 39
  10. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
  11. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 38
  12. 2025T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generationcited by 38
  13. 2025Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modelingcited by 35
  14. 2025FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruningcited by 34
  15. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesiscited by 33
  16. 2024Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficientcited by 32
  17. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  18. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 30
  19. 2025Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generationcited by 30
  20. 2024CAR: Controllable Autoregressive Modeling for Visual Generationcited by 29
  21. 2025Neighboring Autoregressive Modeling for Efficient Visual Generationcited by 29
  22. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokenscited by 27
  23. 2025Next Block Prediction: Video Generation via Semi-Autoregressive Modelingcited by 26
  24. 2025TextAtlas5M: A Large-scale Dataset for Dense Text Image Generationcited by 25
  25. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 25
  26. 2025Perceive, Understand and Restore: Real-World Image Super-Resolution with Autoregressive Multimodal Generative Modelscited by 24
  27. 2025RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulationcited by 24
  28. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 23
  29. 2025FlexVAR: Flexible Visual Autoregressive Modeling without Residual Predictioncited by 22
  30. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  31. 2025Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspectivecited by 22
  32. 2024Customize Your Visual Autoregressive Recipe with Set Autoregressive Modelingcited by 21
  33. 2024ZipAR: Parallel Auto-regressive Image Generation through Spatial Localitycited by 20
  34. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 18
  35. 2025Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Modelscited by 18
  36. 2025Grouped Speculative Decoding for Autoregressive Image Generationcited by 17
  37. 2024TAR3D: Creating High-Quality 3D Assets via Next-Part Predictioncited by 16
  38. 2025ChemMLLM: Chemical Multimodal Large Language Modelcited by 16

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Unified Text-Image Generation with Weakness-Targeted Post-Training
  2. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generation
  3. 2025LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation
  4. 2025LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization
  5. 2026UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
  6. 2026STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation
  7. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  8. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Text