mlsanityData to 14 Oct 2026

CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models

arXiv 2312.06059 ↗First posted 11 Dec 2023cs.CV cs.AI cs.LGcited by 54

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Images produced by text-to-image diffusion models might not always faithfully represent the semantic intent of the provided text prompt, where the model might overlook or entirely fail to produce certain objects. Existing solutions often require customly tailored functions for each of these problems, leading to sub-optimal results, especially for complex prompts. Our work introduces a novel perspective by tackling this challenge in a contrastive context. Our approach intuitively promotes the segregation of objects in attention maps while also maintaining that pairs of related attributes are kept close to each other. We conduct extensive experiments across a wide variety of scenarios, each involving unique combinations of objects, attributes, and scenes. These experiments effectively showcase the versatility, efficiency, and flexibility of our method in working with both latent and pixel-based diffusion models, including Stable Diffusion and Imagen. Moreover, we publicly share our source code to facilitate further research.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 46 of its 51 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

46 of 51 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
  2. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursing20 Jul
  3. 2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesis26 Jun
  4. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis7 Apr
  5. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  6. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  7. 2022Investigating Prompt Engineering in Diffusion Models21 Nov
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  9. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models26 Oct
  10. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  11. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attention10 Oct
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
6 more
  1. 2022Blended Latent Diffusion6 Jun
  2. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  6. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
16 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  3. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  5. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  6. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  7. 2023Zero-shot Image-to-Image Translationcited by 505
  8. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  9. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  10. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  11. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  12. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  13. 2023Teaching CLIP to Count to Tencited by 198
  14. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 162
  15. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  16. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 18; 2025: 24; 2026: 11.

54 papers.

Sort:

By area:

  1. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  2. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 61
  3. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  4. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  5. 2023LIME: Localized Image Editing via Attention Regularization in Diffusion Modelscited by 27
  6. 2024MC2: Multi-concept Guidance for Customized Multi-concept Generationcited by 26
  7. 2025Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformerscited by 20
  8. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  9. 2024GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaborationcited by 17
  10. 2024MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Modelscited by 17
  11. 2024Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjectscited by 16
  12. 2025Towards Transformer-Based Aligned Generation with Self-Coherence Guidancecited by 14
38 more
  1. 2024Progressive Compositionality in Text-to-Image Generative Modelscited by 12
  2. 2024Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Mapscited by 12
  3. 2025LoRAShop: Training-Free Multi-Concept Image Generation and Editing with Rectified Flow Transformerscited by 12
  4. 2024Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generationcited by 11
  5. 2024Contrastive Test-Time Composition of Multiple LoRA Models for Image Generationcited by 10
  6. 2025Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesiscited by 10
  7. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 10
  8. 2025Context-guided Responsible Data Augmentation with Diffusion Modelscited by 9
  9. 2025Image Generation from Contextually-Contradictory Promptscited by 9
  10. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  11. 2025Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Modelscited by 8
  12. 2025VSC: Visual Search Compositional Text-to-Image Diffusion Modelcited by 7
  13. 2025Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioningcited by 7
  14. 2024Isolated Diffusion: Optimizing Multi-Concept Text-to-Image Generation Training-Freely with Isolated Diffusion Guidancecited by 6
  15. 2025Steer Away From Mode Collisions: Improving Composition In Diffusion Modelscited by 5
  16. 2024VitaGlyph: Vitalizing Artistic Typography with Flexible Dual-branch Diffusion Modelscited by 4
  17. 2025Fine-Grained Alignment and Noise Refinement for Compositional Text-to-Image Generationcited by 4
  18. 2025Antithetic Noise in Diffusion Modelscited by 4
  19. 2025Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Imagecited by 3
  20. 2025DOS: Directional Object Separation in Text Embeddings for Multi-Object Image Generationcited by 2
  21. 2025TextGuider: Training-Free Guidance for Text Rendering via Attention Alignmentcited by 2
  22. 2026DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generationcited by 2
  23. 2024SPDiffusion: Semantic Protection Diffusion Models for Multi-concept Text-to-image Generationcited by 1
  24. 2025JEDI: The Force of Jensen-Shannon Divergence in Disentangling Diffusion Modelscited by 1
  25. 2025CountCluster: Training-Free Object Quantity Guidance with Cross-Attention Map Clustering for Text-to-Image Generationcited by 1
  26. 2025InfSplign: Inference-Time Spatial Alignment of Text-to-Image Diffusion Modelscited by 1
  27. 2026EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesiscited by 1
  28. 2026Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfoldscited by 1
  29. 2026T2LSC-Bench: Benchmarking Localized Semantic Control in Text-to-Image Generationcited by 1
  30. 2024ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Imagescited by 0
  31. 2025ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generationcited by 0
  32. 2025ComposeAnything: Composite Object Priors for Text-to-Image Generationcited by 0
  33. 2025Performance Plateaus in Inference-Time Scaling for Text-to-Image Diffusion Without External Modelscited by 0
  34. 2025FOCUS: Optimal Control for Multi-Entity World Modeling in Text-to-Image Generationcited by 0
  35. 2025Synthetic Curriculum Reinforces Compositional Text-to-Image Generationcited by 0
  36. 2026Delta-K: Boosting Multi-Instance Generation via Cross-Attention Augmentationcited by 0
  37. 2026Training-Free Object-Background Compositional T2I via Dynamic Spatial Guidance and Multi-Path Pruningcited by 0
  38. 2026Boosting Text-to-Image Diffusion Models via Core Token Attention-Based Seed Selectioncited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  2. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  3. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models
  4. 2024Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models
  5. 2023The Hidden Language of Diffusion Models
  6. 2026Toward Early Quality Assessment of Text-to-Image Diffusion Models
  7. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decomposition
  8. 2023Semantic Guidance Tuning for Text-To-Image Diffusion Models