# Aligning Text to Image in Diffusion Models is Easier Than You Think

arXiv: https://arxiv.org/abs/2503.08250 · HTML: https://mlsanity.com/paper/2503.08250/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 41 of its 49 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)

## Built on by (most cited first, 22 of 22)

- [Diffuse and Disperse: Image Generation with Representation Regularization](https://mlsanity.com/paper/2506.09027/) (2025)
- [Stepwise Credit Assignment for GRPO on Flow-Matching Models](https://mlsanity.com/paper/2603.28718/) (2026)
- [DreamWorld: Unified World Modeling in Video Generation](https://mlsanity.com/paper/2603.00466/) (2026)
- [DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformers](https://mlsanity.com/paper/2603.04239/) (2026)
- [MATRIX: Mask Track Alignment for Interaction-aware Video Generation](https://mlsanity.com/paper/2510.07310/) (2025)
- [Sprint: Sparse-Dense Residual Fusion for Efficient Diffusion Transformers](https://mlsanity.com/paper/2510.21986/) (2025)
- [Boosting Latent Diffusion Models via Disentangled Representation Alignment](https://mlsanity.com/paper/2601.05823/) (2026)
- [DetailAnywhere: Fashion Detail Generation via Cross-Modal Feature Alignment Distillation](https://mlsanity.com/paper/2607.02220/) (2026)
- [Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation](https://mlsanity.com/paper/2511.01450/) (2025)
- [Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision](https://mlsanity.com/paper/2512.23426/) (2025)
- [MCLR: Improving Conditional Modeling via Inter-Class Likelihood-Ratio Maximization and Unifying Classifier-Free Guidance with Alignment Objectives](https://mlsanity.com/paper/2603.22364/) (2026)
- [Repulsor: Accelerating Generative Modeling with a Contrastive Memory Bank](https://mlsanity.com/paper/2512.08648/) (2025)
- [CORAL: Correspondence Alignment for Improved Virtual Try-On](https://mlsanity.com/paper/2602.17636/) (2026)
- [Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs](https://mlsanity.com/paper/2605.09433/) (2026)
- [DiverseDiT++: Quantifying, Analyzing, and Promoting Representation Diversity in Diffusion Transformers](https://mlsanity.com/paper/2608.03082/) (2026)
- [Align Your Query: Representation Alignment for Multimodality Medical Object Detection](https://mlsanity.com/paper/2510.02789/) (2025)
- [Representation-Regularized Convolutional Audio Transformer for Audio Understanding](https://mlsanity.com/paper/2601.21612/) (2026)
- [Alignment-Guided Score Matching for Text-to-Image Alignment in Diffusion Models](https://mlsanity.com/paper/2605.30038/) (2026)
- [Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models](https://mlsanity.com/paper/2607.02909/) (2026)
- [Introspective Attention Modulation for Safe Text-to-Image Generation](https://mlsanity.com/paper/2607.14945/) (2026)
- [Representation Learning in Diffusion and Flow-based Model: An Application Aspect](https://mlsanity.com/paper/2608.24068/) (2026)
- [Beyond Random Couplings: Contrastive Noise Alignment in Generative Flows](https://mlsanity.com/paper/2609.18488/) (2026)

## Abstract (arXiv)

While recent advancements in generative modeling have significantly improved text-image alignment, some residual misalignment between text and image representations still remains. Some approaches address this issue by fine-tuning models in terms of preference optimization, etc., which require tailored datasets. Orthogonal to these methods, we revisit the challenge from the perspective of representation alignment-an approach that has gained popularity with the success of REPresentation Alignment (REPA). We first argue that conventional text-to-image (T2I) diffusion models, typically trained on paired image and text data (i.e., positive pairs) by minimizing score matching or flow matching losses, is suboptimal from the standpoint of representation alignment. Instead, a better alignment can be achieved through contrastive learning that leverages existing dataset as both positive and negative pairs. To enable efficient alignment with pretrained models, we propose SoftREPA- a lightweight contrastive fine-tuning strategy that leverages soft text tokens for representation alignment. This approach improves alignment with minimal computational overhead by adding fewer than 1M trainable parameters to the pretrained model. Our theoretical analysis demonstrates that our method explicitly increases the mutual information between text and image representations, leading to enhanced semantic consistency. Experimental results across text-to-image generation and text-guided image editing tasks validate the effectiveness of our approach in improving the semantic consistency of T2I generative models.