Pareto-Enhanced Portrait Generation: Vision-Aligned Text Supervision for Alignment, Realism, and Aesthetics
On the map: Diffusion models → Image generation, text-to-image and editing
Abstract (arXiv)
Text-to-image diffusion models often face a severe trilemma in human portrait generation: text-image alignment, photorealism, and human-perceived aesthetics inherently inhibit one another. Supervised Fine-Tuning (SFT) is an effective method for enhancing the photorealism of image generation. However, it often leads to overfitting to the training dataset, corrupts pre-trained image priors, and degrades alignment or aesthetics. To break this bottleneck, we propose a feature supervision paradigm for Multimodal Diffusion Transformers (MM-DiT). Specifically, we introduce a lightweight cross-modal alignment mechanism that implicitly extracts multi-granularity vision-aligned text representations from SigLIP 2 and applies supervision to the image branch of MM-DiT during the training stage, with zero extra inference overhead. Our method injects vision-aligned text guidance while preserving the base model's original generalization, avoiding degradation caused by SFT. Furthermore, our method directly mines implicit multi-granularity aesthetic signals from pre-trained vision foundation models to optimize human-perceived aesthetics. Extensive experiments on MM-DiTs show that our method pushes the Pareto frontier and achieves synergistic improvements across text-image alignment, photorealism, and human-perceived aesthetics.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 15 of its 27 references that are in the map. First posted on arXiv 20 May 2026.
Builds on
15 of 27 references.
Closest ancestors · 1
Same direction, newest first
Other references · 14
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2020Denoising Diffusion Implicit Modelscited by 9,655
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
- 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models
- 2023Improving Compositional Text-to-image Generation with Large Vision-Language Models
- 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
- 2025CompArt: Operationalizing Aesthetic Alignment in Text-to-Image Generation via Principles of Art
- 2023Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression
- 2025CritiFusion: Semantic Critique and Spectral Alignment for Faithful Text-to-Image Generation
- 2025UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception
- 2025Aligning Text to Image in Diffusion Models is Easier Than You Think