mlsanityData to 7 Oct 2026

Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models

arXiv 2304.01515 ↗First posted 4 Apr 2023cs.LG cs.CL cs.CVcited by 6

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Token-based masked generative models are gaining popularity for their fast inference time with parallel decoding. While recent token-based approaches achieve competitive performance to diffusion-based models, their generation performance is still suboptimal as they sample multiple tokens simultaneously without considering the dependence among them. We empirically investigate this problem and propose a learnable sampling model, Text-Conditioned Token Selection (TCTS), to select optimal tokens via localized supervision with text information. TCTS improves not only the image quality but also the semantic alignment of the generated images with the given texts. To further improve the image quality, we introduce a cohesive sampling strategy, Frequency Adaptive Sampling (FAS), to each group of tokens divided according to the self-attention maps. We validate the efficacy of TCTS combined with FAS with various generative tasks, demonstrating that it significantly outperforms the baselines in image-text alignment and image quality. Our text-conditioned sampling framework further reduces the original inference time by more than 50% without modifying the original generative model.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 45 of its 49 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

45 of 49 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
33 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  6. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  9. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  10. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  11. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  12. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  13. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  14. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  15. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  16. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  17. 2022MAGVIT: Masked Generative Video Transformercited by 341
  18. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  19. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  20. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 151
  21. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  22. 2021Instance-Conditioned GANcited by 119
  23. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  24. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  25. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesiscited by 75
  26. 2021Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codescited by 71
  27. 2022Improved Masked Image Generation with Token-Criticcited by 63
  28. 2022Improved Vector Quantized Diffusion Modelscited by 62
  29. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  30. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  31. 2022Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformercited by 30
  32. 2017Fine-grained Visual-textual Representation Learningcited by 27
  33. 2022Lafite2: Few-shot Text-to-Image Generationcited by 13

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 2.

6 papers.

  1. 2024Language-driven Grasp Detectioncited by 26
  2. 2023A Pytorch Reproduction of Masked Generative Image Transformercited by 25
  3. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
  4. 2023STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignmentcited by 1
  5. 2025UNCAGE: Contrastive Attention Guidance for Masked Generative Transformers in Text-to-Image Generationcited by 0
  6. 2025Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Language-Guided Image Tokenization for Generation
  2. 2023P+: Extended Textual Conditioning in Text-to-Image Generation
  3. 2025CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
  4. 2024Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer
  5. 2025Designing Instance-Level Sampling Schedules via REINFORCE with James-Stein Shrinkage
  6. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervision
  7. 2026Let Triggers Control: Frequency-Aware Dropout for Effective Token Control
  8. 2024STAR: Scale-wise Text-conditioned AutoRegressive image generation