mlsanityData to 14 Oct 2026

Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation

arXiv 2401.05675 ↗First posted 11 Jan 2024cs.CVcited by 36

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

also in 1 more
Abstract (arXiv)

Recent works have demonstrated that using reinforcement learning (RL) with multiple quality rewards can improve the quality of generated images in text-to-image (T2I) generation. However, manually adjusting reward weights poses challenges and may cause over-optimization in certain metrics. To solve this, we propose Parrot, which addresses the issue through multi-objective optimization and introduces an effective multi-reward optimization strategy to approximate Pareto optimal. Utilizing batch-wise Pareto optimal selection, Parrot automatically identifies the optimal trade-off among different rewards. We use the novel multi-reward optimization algorithm to jointly optimize the T2I model and a prompt expansion network, resulting in significant improvement of image quality and also allow to control the trade-off of different rewards using a reward related prompt during inference. Furthermore, we introduce original prompt-centered guidance at inference time, ensuring fidelity to user input after prompt expansion. Extensive experiments and a user study validate the superiority of Parrot over several baselines across various quality criteria, including aesthetics, human preference, text-image alignment, and image sentiment.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 58 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

48 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  2. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar
  3. 2023Aligning Text-to-Image Models using Human Feedback23 Feb

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  10. 2020Denoising Diffusion Implicit Modelscited by 9,781
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
33 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  7. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  8. 2023PaLM 2 Technical Reportcited by 1,162
  9. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,081
  10. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  11. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  12. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  13. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  14. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  15. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  16. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  17. 2019KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessmentcited by 355
  18. 2022MAGVIT: Masked Generative Video Transformercited by 346
  19. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  20. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  21. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  22. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  23. 2019From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Qualitycited by 223
  24. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  25. 2016Multi-Objective Deep Reinforcement Learningcited by 83
  26. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  27. 2023VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretrainingcited by 65
  28. 2022Pareto Set Learning for Expensive Multi-Objective Optimizationcited by 51
  29. 2023The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusioncited by 41
  30. 2022Shifted Diffusion for Text-to-image Generationcited by 37
  31. 2023Soundini: Sound-Guided Diffusion for Natural Video Editingcited by 14
  32. 2023Learning Profitable NFT Image Diffusions via Multiple Visual-Policy Guided Reinforcement Learningcited by 7
  33. 2023The Emotions of the Crowd: Learning Image Sentiment from Tweets via Cross-modal Distillationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 18; 2026: 9.

36 papers.

Sort:

By area:

  1. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  2. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 41
  5. 2025MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generationcited by 23
  6. 2024VersaT2I: Improving Text-to-Image Models with Versatile Rewardcited by 17
  7. 2024Towards Reliable Advertising Image Generation Using Human Feedbackcited by 16
  8. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 12
  9. 2024Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learningcited by 11
  10. 2024Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learningcited by 10
  11. 2025Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generationcited by 9
  12. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
24 more
  1. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  2. 2025Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewardscited by 8
  3. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 7
  4. 2025iDSE: Navigating Design Space Exploration in High-Level Synthesis Using LLMscited by 7
  5. 2026coDrawAgents: A Multi-Agent Dialogue Framework for Compositional Image Generationcited by 7
  6. 2025LongAnimation: Long Animation Generation with Dynamic Global-Local Memorycited by 6
  7. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  8. 2024YaART: Yet Another ART Rendering Technologycited by 5
  9. 2025MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generationcited by 5
  10. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Modelscited by 4
  11. 2025POET: Preference Optimization for Enhanced Text-to-Image Generationcited by 4
  12. 2025Aligned Multi Objective Optimizationcited by 3
  13. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RLcited by 3
  14. 2025Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignmentcited by 3
  15. 2025Subjective Camera 1.0: Bridging Human Cognition and Visual Reconstruction through Sequence-Aware Sketch-Guided Diffusioncited by 3
  16. 2026ParetoSlider: Diffusion Models Post-Training for Continuous Reward Controlcited by 2
  17. 2025InstructEngine: Instruction-driven Text-to-Image Alignmentcited by 1
  18. 2026Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributescited by 1
  19. 2026POCA: Pareto-Optimal Curriculum Alignment for Visual Text Generationcited by 1
  20. 2025A Self-explainable Model of Long Time Series by Extracting Informative Structured Causal Patternscited by 0
  21. 2026PoM: A Linear-Time Replacement for Attention with the Polynomial Mixercited by 0
  22. 2026Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarizationcited by 0
  23. 2026DyCoRM: Dynamic Criterion-Aware Reward Modeling for Text-to-Image Generationcited by 0
  24. 2026OuroReward: Sequential Reward Scheduling for Reinforcement Learning in Text-to-3D Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generation
  2. 2026Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
  3. 2026RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
  4. 2026PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning
  5. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models
  6. 2025MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency
  7. 2024Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning
  8. 2024Dynamic Prompt Optimizing for Text-to-Image Generation