Data to 7 Oct 2026

CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching

arXiv 2404.03653 ↗First posted 4 Apr 2024cs.CV cs.AI cs.CLcited by 60

On the map: Diffusion models → Image generation, text-to-image and editing

also in 1 more

LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. The root reason behind the misalignment has not been extensively investigated. We observe that the misalignment is caused by inadequate token attention activation. We further attribute this phenomenon to the diffusion model's insufficient condition utilization, which is caused by its training paradigm. To address the issue, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with an image-to-text concept matching mechanism. We leverage an image captioning model to measure image-to-text alignment and guide the diffusion model to revisit ignored tokens. A novel attribute concentration module is also proposed to address the attribute binding problem. Without any image or human preference data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL. Extensive experiments show that CoMat-SDXL significantly outperforms the baseline model SDXL in two text-to-image alignment benchmarks and achieves start-of-the-art performance.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 70 of its 94 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

70 of 94 references.

Closest ancestors · 23

Same direction, newest first

11 more

Other references · 47

2017Attention Is All You Needcited by 59,932
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
2023GPT-4 Technical Reportcited by 18,132
2014Microsoft COCO: Common Objects in Contextcited by 17,498
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023Visual Instruction Tuningcited by 7,792
2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310

35 more

2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
2021Zero-Shot Text-to-Image Generationcited by 4,276
2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
2023Training Diffusion Models with Reinforcement Learningcited by 939
2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
2023Adversarial Diffusion Distillationcited by 741
2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
2023Segment Everything Everywhere All at Oncecited by 546
2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
2023Personalize Segment Anything Model with One Shotcited by 250
2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
2021Simple multi-dataset detectioncited by 126
2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
2024Be-Your-Outpainter: Mastering Video Outpainting through Input-Specific Adaptationcited by 29
2023Stable Diffusion is Unstablecited by 27
2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 30; 2026: 12.

60 papers.

Sort:

By area:

2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
2024Phased Consistency Modelscited by 83
2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 35
2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33

38 more

2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 29
2024EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLMcited by 25
2024Robust CLIP-Based Detector for Exposing Diffusion Model-Generated Imagescited by 20
2025D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samplescited by 14
2025Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimizationcited by 12
2025Dual-Process Image Generationcited by 12
2024Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generationcited by 11
2024Mask Approximation Net: A Novel Diffusion Model Approach for Remote Sensing Change Captioningcited by 10
2024Text-to-Image Synthesis: A Decade Surveycited by 9
2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 8
2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8
2025Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMscited by 8
2024CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Modelscited by 7
2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewardscited by 6
2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generationcited by 6
2025Geometrical Properties of Text Token Embeddings for Strong Semantic Binding in Text-to-Image Generationcited by 5
2026Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generationcited by 5
2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesiscited by 4
2025Detail++: Training-Free Detail Enhancer for T2I Diffusion Modelscited by 4
2025Guiding What Not to Generate: Automated Negative Prompting for Text-Image Alignmentcited by 4
2025IMG: Calibrating Diffusion Models via Implicit Multimodal Guidancecited by 3
2025CalibCLIP: Contextual Calibration of Dominant Semantics for Text-Driven Image Retrievalcited by 3
2025AgeBooth: Controllable Facial Aging and Rejuvenation via Diffusion Modelscited by 3
2025Bridging Fidelity-Reality with Controllable One-Step Diffusion for Image Super-Resolutioncited by 3
2024Diffusion-RSCC: Diffusion Probabilistic Model for Change Captioning in Remote Sensing Imagescited by 2
2025CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Explorationcited by 2
2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairscited by 2
2025DOS: Directional Object Separation in Text Embeddings for Multi-Object Image Generationcited by 2
2025Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generationcited by 2
2026Selective Fine-Tuning for Targeted and Robust Concept Unlearningcited by 2
2024Low-Biased General Annotated Dataset Generationcited by 1
2025Coffee: Controllable Diffusion Fine-tuningcited by 1
2026Say Cheese! Detail-Preserving Portrait Collection Generation via Natural Language Editscited by 1
2026Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformerscited by 1
2025ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generationcited by 0
2025ComposeAnything: Composite Object Priors for Text-to-Image Generationcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way