mlsanityData to 13 Oct 2026

TokenCompose: Text-to-Image Diffusion with Token-level Supervision

arXiv 2312.03626 ↗First posted 6 Dec 2023cs.CVcited by 40

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We present TokenCompose, a Latent Diffusion Model for text-to-image generation that achieves enhanced consistency between user-specified text prompts and model-generated images. Despite its tremendous success, the standard denoising process in the Latent Diffusion Model takes text prompts as conditions only, absent explicit constraint for the consistency between the text prompts and the image contents, leading to unsatisfactory results for composing multiple object categories. TokenCompose aims to improve multi-category instance composition by introducing the token-wise consistency terms between the image content and object segmentation maps in the finetuning stage. TokenCompose can be applied directly to the existing training pipeline of text-conditioned diffusion models without extra human labeling information. By finetuning Stable Diffusion, the model exhibits significant improvements in multi-category instance composition and enhanced photorealism for its generated images. Project link: https://mlpc-ucsd.github.io/TokenCompose

Key facts. Cited by 40 papers in the map (data through 13 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 68 of its 80 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

68 of 80 references.

Closest ancestors · 22

Same direction, newest first

  1. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation25 Oct
  2. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment15 Jun
  3. 2023Controllable Text-to-Image Generation with GPT-429 May
  4. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths29 May
  5. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models11 Apr
  6. 2023Your Diffusion Model is Secretly a Zero-Shot Classifier28 Mar
  7. 2023Text-to-Image Diffusion Models are Zero-Shot Classifiers27 Mar
  8. 2023DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery17 Mar
  9. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception3 Mar
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  11. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  12. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
10 more
  1. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  2. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  3. 2022DiffusionDet: Diffusion Model for Object Detection17 Nov
  4. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts27 Oct
  5. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attention10 Oct
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models8 Feb
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  8. 2020Denoising Diffusion Implicit Modelscited by 9,718
  9. 2023Segment Anythingcited by 8,195
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,000
  12. 2022Classifier-Free Diffusion Guidancecited by 5,785
34 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,299
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,421
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,407
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,344
  8. 2015Variational Inference with Normalizing Flowscited by 2,163
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,355
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,267
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,068
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 955
  13. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 692
  14. 2023Aligning Text-to-Image Models using Human Feedbackcited by 420
  15. 2023Segment Anything in High Qualitycited by 374
  16. 2023Scaling Open-Vocabulary Object Detectioncited by 337
  17. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  18. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  19. 2022Visual Spatial Reasoningcited by 290
  20. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 255
  21. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  22. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  23. 2021Simple multi-dataset detectioncited by 126
  24. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 97
  25. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  26. 2023Open-vocabulary Object Segmentation with Diffusion Modelscited by 82
  27. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  28. 2023Bayesian Flow Networkscited by 63
  29. 2023OVTrack: Open-Vocabulary Multiple Object Trackingcited by 53
  30. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  31. 2023SLiMe: Segment Like Mecited by 34
  32. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  33. 2017Wasserstein Introspective Neural Networkscited by 18
  34. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Modelscited by 16

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 13; 2025: 16; 2026: 11.

40 papers.

Sort:

By area:

  1. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  2. 2024MC2: Multi-concept Guidance for Customized Multi-concept Generationcited by 26
  3. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  4. 2024Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Functioncited by 19
  5. 2025Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformerscited by 19
  6. 2024Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjectscited by 16
  7. 2025High-Order Matching for One-Step Shortcut Diffusion Modelscited by 15
  8. 2025Circuit Complexity Bounds for Visual Autoregressive Modelcited by 10
  9. 2025On Computational Limits of FlowAR Models: Expressivity and Efficiencycited by 10
  10. 2025Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Modelscited by 10
  11. 2025OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlapscited by 10
  12. 2024Text-to-Image Synthesis: A Decade Surveycited by 9
28 more
  1. 2024The Art of Deception: Color Visual Illusions and Diffusion Modelscited by 9
  2. 2025Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policiescited by 9
  3. 2025TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Visioncited by 9
  4. 2025T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Modelscited by 9
  5. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  6. 2024CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Modelscited by 7
  7. 2024Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generationcited by 4
  8. 2024Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentionscited by 4
  9. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesiscited by 4
  10. 2025HOFAR: High-Order Augmentation of Flow Autoregressive Transformerscited by 4
  11. 2025Robot Motion Planning using One-Step Diffusion with Noise-Optimized Approximate Motionscited by 3
  12. 2025LoRAtorio: An intrinsic approach to LoRA Skill Compositioncited by 3
  13. 2026HybridStitch: Pixel and Timestep Level Model Stitching for Diffusion Accelerationcited by 2
  14. 2026Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimizationcited by 2
  15. 2024SPDiffusion: Semantic Protection Diffusion Models for Multi-concept Text-to-image Generationcited by 1
  16. 2026RegionRoute: Regional Style Transfer with Diffusion Modelcited by 1
  17. 2026EPIG: Emotion-Based Prompting for Personalised Image Generationcited by 1
  18. 2025ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generationcited by 0
  19. 2025ComposeAnything: Composite Object Priors for Text-to-Image Generationcited by 0
  20. 2025Controllable Coupled Image Generation via Diffusion Modelscited by 0
  21. 2025FOCUS: Optimal Control for Multi-Entity World Modeling in Text-to-Image Generationcited by 0
  22. 2026PureCC: Pure Learning for Text-to-Image Concept Customizationcited by 0
  23. 2026Let Triggers Control: Frequency-Aware Dropout for Effective Token Controlcited by 0
  24. 2026Training-Free Object-Background Compositional T2I via Dynamic Spatial Guidance and Multi-Path Pruningcited by 0
  25. 2026ELDiff: When Evidential Learning Meets Text-to-Image Diffusioncited by 0
  26. 2026Attention Dynamics in Diffusion Models: A Visual Analytics Framework for Human-AI Collaborationcited by 0
  27. 2026Diffusion Image Editing via Asynchronous Token Decodingcited by 0
  28. 2026Paint-Anything: Unified Any-Color Control for Image Generation and Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models
  2. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  3. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
  5. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds
  6. 2022Multi-Concept Customization of Text-to-Image Diffusion
  7. 2025CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Models
  8. 2024Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback