mlsanityData to 7 Oct 2026

Muse: Text-To-Image Generation via Masked Generative Transformers

arXiv 2301.00704 ↗First posted 2 Jan 2023cs.CV cs.AI cs.LGcited by 640

Not on the map. Why?

Cited by 256 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io

Key facts. Cited by 640 papers in the map (data through 7 Oct 2026). Builds on 69 of its 87 references that are in the map. First posted on arXiv 2 Jan 2023.

Builds on

69 of 87 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2020Denoising Diffusion Implicit Modelscited by 9,655
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  13. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  14. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  15. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  16. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  17. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  18. 2022Video Diffusion Modelscited by 2,280
  19. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  20. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  21. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  22. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  23. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  24. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  25. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  26. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  27. 2021Variational Diffusion Modelscited by 1,145
  28. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  29. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  30. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  31. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  32. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  33. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  34. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  35. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  36. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  37. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  38. 2021GAN Inversion: A Surveycited by 366

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 199; 2024: 215; 2025: 163; 2026: 63.

640 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  2. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  3. 2025Large Language Diffusion Modelscited by 872
  4. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  6. 2023Adversarial Diffusion Distillationcited by 741
  7. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  8. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  9. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  11. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  12. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
38 more
  1. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  2. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  3. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  4. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  5. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  6. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  7. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  8. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  9. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  10. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  11. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  12. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  13. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  14. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  15. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  16. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  17. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  18. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  19. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 303
  20. 2024Discrete Flow Matchingcited by 301
  21. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 298
  22. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  23. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  24. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  25. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  26. 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
  27. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  28. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  29. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  30. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  31. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  32. 2023A Cookbook of Self-Supervised Learningcited by 248
  33. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  34. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  35. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  36. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Modelscited by 240
  37. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  38. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230

50 most cited of 640 All 640 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MAGVLT: Masked Generative Vision-and-Language Transformer
  2. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  3. 2024aMUSEd: An Open MUSE Reproduction
  4. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model
  5. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation
  6. 2025LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization
  7. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generation
  8. 2026Text-to-Image Models Need Less from Text Encoders Than You Think