mlsanityData to 7 Oct 2026

Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignment

arXiv 2507.19002 ↗First posted 25 Jul 2025cs.CVcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Contemporary image generation systems have achieved high fidelity and superior aesthetic quality beyond basic text-image alignment. However, existing evaluation frameworks have failed to evolve in parallel. This study reveals that human preference reward models fine-tuned based on CLIP and BLIP architectures have inherent flaws: they inappropriately assign low scores to images with rich details and high aesthetic value, creating a significant discrepancy with actual human aesthetic preferences. To address this issue, we design a novel evaluation score, ICT (Image-Contained-Text) score, that achieves and surpasses the objectives of text-image alignment by assessing the degree to which images represent textual content. Building upon this foundation, we further train an HP (High-Preference) score model using solely the image modality to enhance image aesthetics and detail quality while maintaining text-image alignment. Experiments demonstrate that the proposed evaluation model improves scoring accuracy by over 10% compared to existing methods, and achieves significant results in optimizing state-of-the-art text-to-image models. This research provides theoretical and empirical support for evolving image generation technology toward higher-order human aesthetic preferences. Code is available at https://github.com/BarretBa/ICTHP.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 45 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

34 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  3. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
19 more
  1. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  2. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  3. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  4. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  5. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  6. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  7. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  8. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  9. 2024InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generationcited by 166
  10. 2024Dynamic Prompt Optimizing for Text-to-Image Generationcited by 45
  11. 2024STAR: Scale-wise Text-conditioned AutoRegressive image generationcited by 28
  12. 2023Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Samplingcited by 19
  13. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  14. 2024Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspectivecited by 9
  15. 2025V2Flow: Unifying Visual Tokenization and Large Language Model Vocabularies for Autoregressive Image Generationcited by 6
  16. 2024Learning Invariant Causal Mechanism from Vision-Language Modelscited by 3
  17. 2024Uniform Attention Maps: Boosting Image Fidelity in Reconstruction and Editingcited by 2
  18. 2024StyleInject: Parameter Efficient Tuning of Text-to-Image Diffusion Modelscited by 1
  19. 2025On the Out-of-Distribution Generalization of Self-Supervised Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preferencecited by 30
  2. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 30
  3. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generationcited by 11
  4. 2026Fine-grained Image Aesthetic Assessment: Learning Discriminative Scores from Relative Rankscited by 5
  5. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  6. 2026Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Modelscited by 4
  7. 2025PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generationcited by 3
  8. 2025SafeEditor: Unified MLLM for Efficient Post-hoc T2I Safety Editingcited by 2
  9. 2025PreferThinker: Reasoning-based Personalized Image Preference Assessmentcited by 0
  10. 2026DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generationcited by 0
  11. 2026Efficient Personalization of Generative User Interfacescited by 0
  12. 2026Pareto-Guided Optimal Transport for Multi-Reward Alignmentcited by 0
2 more
  1. 2026Biological Reasoning-Informed Regression for Interpretable Regulatory DNA Activity Predictioncited by 0
  2. 2026Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025What Makes a Good Generated Image? Investigating Human and Multimodal LLM Image Preference Alignment
  2. 2025CompArt: Operationalizing Aesthetic Alignment in Text-to-Image Generation via Principles of Art
  3. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation
  4. 2023Aligning Text-to-Image Models using Human Feedback
  5. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
  6. 2026Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards
  7. 2025CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
  8. 2025YINYANG-ALIGN: Benchmarking Contradictory Objectives and Proposing Multi-Objective Optimization based DPO for Text-to-Image Alignment