mlsanityData to 14 Oct 2026

Efficient Personalization of Generative Models via Optimal Experimental Design

arXiv 2512.19057 ↗First posted 22 Dec 2025cs.LG cs.IT math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

Preference learning from human feedback has the ability to align generative models with the needs of end-users. Human feedback is costly and time-consuming to obtain, which creates demand for data-efficient query selection methods. This work presents a novel approach that leverages optimal experimental design to ask humans the most informative preference queries, from which we can elucidate the latent reward function modeling user preferences efficiently. We formulate the problem of preference query selection as the one that maximizes the information about the underlying latent preference model. We show that this problem has a convex optimization formulation, and introduce a statistically and computationally efficient algorithm ED-PBRL that is supported by theoretical guarantees and can efficiently construct structured queries such as images or text. We empirically present the proposed framework by personalizing a text-to-image generative model to user-specific styles, showing that it requires less preference queries compared to random query selection.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 22 Dec 2025.

Builds on

34 of 54 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
22 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  4. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  6. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  8. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  9. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  10. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  11. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  12. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  13. 2021Reward is enough for convex MDPscited by 84
  14. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  15. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
  16. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  17. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  18. 2022Active Exploration for Inverse Reinforcement Learningcited by 27
  19. 2021Information Directed Reward Learning for Reinforcement Learningcited by 24
  20. 2022Active Exploration via Experiment Design in Markov Chainscited by 19
  21. 2024Transition Constrained Bayesian Optimization via Markov Decision Processescited by 7
  22. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3

Built on by

Not cited yet.