mlsanityData to 7 Oct 2026

Pareto-Enhanced Portrait Generation: Vision-Aligned Text Supervision for Alignment, Realism, and Aesthetics

arXiv 2605.20640 ↗First posted 20 May 2026cs.CV cs.AIcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models often face a severe trilemma in human portrait generation: text-image alignment, photorealism, and human-perceived aesthetics inherently inhibit one another. Supervised Fine-Tuning (SFT) is an effective method for enhancing the photorealism of image generation. However, it often leads to overfitting to the training dataset, corrupts pre-trained image priors, and degrades alignment or aesthetics. To break this bottleneck, we propose a feature supervision paradigm for Multimodal Diffusion Transformers (MM-DiT). Specifically, we introduce a lightweight cross-modal alignment mechanism that implicitly extracts multi-granularity vision-aligned text representations from SigLIP 2 and applies supervision to the image branch of MM-DiT during the training stage, with zero extra inference overhead. Our method injects vision-aligned text guidance while preserving the base model's original generalization, avoiding degradation caused by SFT. Furthermore, our method directly mines implicit multi-granularity aesthetic signals from pre-trained vision foundation models to optimize human-perceived aesthetics. Extensive experiments on MM-DiTs show that our method pushes the Pareto frontier and achieves synergistic improvements across text-image alignment, photorealism, and human-perceived aesthetics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 15 of its 27 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

15 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  12. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
2 more
  1. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  2. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models
  2. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Models
  3. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
  4. 2025CompArt: Operationalizing Aesthetic Alignment in Text-to-Image Generation via Principles of Art
  5. 2023Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression
  6. 2025CritiFusion: Semantic Critique and Spectral Alignment for Faithful Text-to-Image Generation
  7. 2025UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception
  8. 2025Aligning Text to Image in Diffusion Models is Easier Than You Think