# Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignment

arXiv: https://arxiv.org/abs/2507.19002 · HTML: https://mlsanity.com/paper/2507.19002/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Directly Fine-Tuning Diffusion Models on Differentiable Rewards](https://mlsanity.com/paper/2309.17400/) (2023)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://mlsanity.com/paper/2303.14420/) (2023)
- [LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models](https://mlsanity.com/paper/2505.19223/) (2025)
- [Controlling Text-to-Image Diffusion by Orthogonal Finetuning](https://mlsanity.com/paper/2306.07280/) (2023)
- [Optimizing Prompts for Text-to-Image Generation](https://mlsanity.com/paper/2212.09611/) (2022)
- [InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation](https://mlsanity.com/paper/2404.02733/) (2024)
- [Dynamic Prompt Optimizing for Text-to-Image Generation](https://mlsanity.com/paper/2404.04095/) (2024)
- [STAR: Scale-wise Text-conditioned AutoRegressive image generation](https://mlsanity.com/paper/2406.10797/) (2024)
- [Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Sampling](https://mlsanity.com/paper/2308.01850/) (2023)
- [Aligning Few-Step Diffusion Models with Dense Reward Difference Learning](https://mlsanity.com/paper/2411.11727/) (2024)
- [Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective](https://mlsanity.com/paper/2410.12816/) (2024)
- [V2Flow: Unifying Visual Tokenization and Large Language Model Vocabularies for Autoregressive Image Generation](https://mlsanity.com/paper/2503.07493/) (2025)

## Built on by (most cited first, 14 of 14)

- [JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization](https://mlsanity.com/paper/2511.23002/) (2025)
- [Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference](https://mlsanity.com/paper/2509.06942/) (2025)
- [The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation](https://mlsanity.com/paper/2511.20256/) (2025)
- [Fine-grained Image Aesthetic Assessment: Learning Discriminative Scores from Relative Ranks](https://mlsanity.com/paper/2603.03907/) (2026)
- [TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models](https://mlsanity.com/paper/2601.08011/) (2026)
- [Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Models](https://mlsanity.com/paper/2603.15724/) (2026)
- [PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generation](https://mlsanity.com/paper/2512.06020/) (2025)
- [SafeEditor: Unified MLLM for Efficient Post-hoc T2I Safety Editing](https://mlsanity.com/paper/2510.24820/) (2025)
- [PreferThinker: Reasoning-based Personalized Image Preference Assessment](https://mlsanity.com/paper/2511.00609/) (2025)
- [DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation](https://mlsanity.com/paper/2602.23438/) (2026)
- [Efficient Personalization of Generative User Interfaces](https://mlsanity.com/paper/2604.09876/) (2026)
- [Pareto-Guided Optimal Transport for Multi-Reward Alignment](https://mlsanity.com/paper/2605.13155/) (2026)
- [Biological Reasoning-Informed Regression for Interpretable Regulatory DNA Activity Prediction](https://mlsanity.com/paper/2606.08147/) (2026)
- [Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation](https://mlsanity.com/paper/2607.11886/) (2026)

## Abstract (arXiv)

Contemporary image generation systems have achieved high fidelity and superior aesthetic quality beyond basic text-image alignment. However, existing evaluation frameworks have failed to evolve in parallel. This study reveals that human preference reward models fine-tuned based on CLIP and BLIP architectures have inherent flaws: they inappropriately assign low scores to images with rich details and high aesthetic value, creating a significant discrepancy with actual human aesthetic preferences. To address this issue, we design a novel evaluation score, ICT (Image-Contained-Text) score, that achieves and surpasses the objectives of text-image alignment by assessing the degree to which images represent textual content. Building upon this foundation, we further train an HP (High-Preference) score model using solely the image modality to enhance image aesthetics and detail quality while maintaining text-image alignment. Experiments demonstrate that the proposed evaluation model improves scoring accuracy by over 10% compared to existing methods, and achieves significant results in optimizing state-of-the-art text-to-image models. This research provides theoretical and empirical support for evolving image generation technology toward higher-order human aesthetic preferences. Code is available at https://github.com/BarretBa/ICTHP.