mlsanityData to 14 Oct 2026

Learning Parametric Distributions from Samples and Preferences

arXiv 2505.23557 ↗First posted 29 May 2025stat.ML cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent advances in language modeling have underscored the role of preference feedback in enhancing model performance. This paper investigates the conditions under which preference feedback improves parameter estimation in classes of continuous parametric distributions. In our framework, the learner observes pairs of samples from an unknown distribution along with their relative preferences depending on the same unknown parameter. We show that preference-based M-estimators achieve a better asymptotic variance than sample-only M-estimators, further improved by deterministic preferences. Leveraging the hard constraints revealed by deterministic preferences, we propose an estimator achieving an estimation error scaling of O(1/n) -- a significant improvement over the Theta(1/√n) rate attainable with samples alone. Next, we establish a lower bound that matches this accelerated rate; up to dimension and problem-dependent constants. While the assumptions underpinning our analysis are restrictive, they are satisfied by notable cases such as Gaussian or Laplace distributions for preferences based on the log-probability reward.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 52 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

28 of 52 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025The Differences Between Direct Alignment Algorithms are a Blur3 Feb
  2. 2025Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspective30 Jan
  3. 2024ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood14 Sep
  4. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  8. 2023Nash Learning from Human Feedbackcited by 224
  9. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  10. 2019The Lean mathematical librarycited by 134
  11. 2020Improved Optimistic Algorithms for Logistic Banditscited by 118
  12. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
8 more
  1. 2015Parallelizing the dual revised simplex methodcited by 89
  2. 2015Simple, Robust and Optimal Ranking from Pairwise Comparisonscited by 70
  3. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 61
  4. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  5. 2022JuMP 1.0: Recent improvements to a modeling language for mathematical optimizationcited by 37
  6. 2014Minimax-optimal Inference from Partial Rankingscited by 24
  7. 2017Minimax Rates and Efficient Algorithms for Noisy Sortingcited by 15
  8. 2024Learning Linear Utility Functions From Pairwise Comparison Queriescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learning Distributions from Samples and Qualitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A density estimation perspective on learning from pairwise human preferences
  2. 2025Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning
  3. 2024Preference-Guided Reflective Sampling for Aligning Language Models
  4. 2025Judging with Confidence: Calibrating Autoraters to Preference Distributions
  5. 2025Offline Preference Optimization via Maximum Marginal Likelihood Estimation
  6. 2025Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
  7. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains
  8. 2024Improving Context-Aware Preference Modeling for Language Models