mlsanityData to 14 Oct 2026

Preference Adaptive and Sequential Text-to-Image Generation

arXiv 2412.10419 ↗First posted 10 Dec 2024cs.CV cs.AI cs.CLcited by 8

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

We address the problem of interactive text-to-image (T2I) generation, designing a reinforcement learning (RL) agent which iteratively improves a set of generated images for a user through a sequence of prompt expansions. Using human raters, we create a novel dataset of sequential preferences, which we leverage, together with large-scale open-source (non-sequential) datasets. We construct user-preference and user-choice models using an EM strategy and identify varying user preference types. We then leverage a large multimodal language model (LMM) and a value-based RL approach to suggest an adaptive and diverse slate of prompt expansions to the user. Our Preference Adaptive and Sequential Text-to-image Agent (PASTA) extends T2I models with adaptive multi-turn capabilities, fostering collaborative co-creation and addressing uncertainty or underspecification in a user's intent. We evaluate PASTA using human raters, showing significant improvement compared to baseline methods. We also open-source our sequential rater dataset and simulated user-rater interactions to support future research in user-centric multi-turn T2I systems.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 68 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

43 of 68 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
31 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  4. 2022Diffusion Models in Vision: A Surveycited by 1,015
  5. 2020Mastering Atari with Discrete World Modelscited by 1,007
  6. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  8. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  9. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  10. 2019Model-Based Reinforcement Learning for Ataricited by 502
  11. 2016Learning to reinforcement learncited by 377
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  13. 2022Transformers are Sample-Efficient World Modelscited by 328
  14. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 287
  15. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  16. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  17. 2015Contextual Markov Decision Processescited by 220
  18. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  19. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  20. 2023Rich Human Feedback for Text-to-Image Generationcited by 141
  21. 2023Matryoshka Diffusion Modelscited by 72
  22. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
  23. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  24. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 35
  25. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  26. 2023FABRIC: Personalizing Diffusion Models with Iterative Feedbackcited by 19
  27. 2024ViPer: Visual Personalization of Generative Models via Individual Preference Learningcited by 17
  28. 2024Embedding-Aligned Language Modelscited by 9
  29. 2023A Diffusion-based Method for Multi-turn Compositional Image Generationcited by 4
  30. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
  31. 2024Taming Text-to-Image Synthesis for Novices: User-centric Prompt Generation via Multi-turn Guidancecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learningcited by 23
  2. 2025Personalized Reward Modeling for Text-to-Image Generationcited by 4
  3. 2026Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generationcited by 3
  4. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  5. 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 1
  6. 2026Visual Personalization Turing Testcited by 0
  7. 2026Bridging the Intention-Expression Gap: Aligning Multi-Dimensional Preferences via Hierarchical Relevance Feedback in Text-to-Image Diffusioncited by 0
  8. 2026SegBanana: Steering Unified Multimodal Models into Medical Segmenterscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Instant Preference Alignment for Text-to-Image Diffusion Models
  2. 2024Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation
  3. 2025Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Preference Understanding
  4. 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewriting
  5. 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
  6. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences
  7. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration
  8. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generation