# Learning Distributions from Samples and Qualities

arXiv: https://arxiv.org/abs/2610.17956 · HTML: https://mlsanity.com/paper/2610.17956/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 67 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)
- [Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons](https://mlsanity.com/paper/2301.11270/) (2023)
- [Generalized Preference Optimization: A Unified Approach to Offline Alignment](https://mlsanity.com/paper/2402.05749/) (2024)
- [MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models](https://mlsanity.com/paper/2406.06046/) (2024)
- [Simple, Robust and Optimal Ranking from Pairwise Comparisons](https://mlsanity.com/paper/1512.08949/) (2015)
- [Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence](https://mlsanity.com/paper/1505.01462/) (2015)
- [JuMP 1.0: Recent improvements to a modeling language for mathematical optimization](https://mlsanity.com/paper/2206.03866/) (2022)
- [Refined Error Bounds for Several Learning Algorithms](https://mlsanity.com/paper/1512.07146/) (2015)
- [Minimax-optimal Inference from Partial Rankings](https://mlsanity.com/paper/1406.5638/) (2014)
- [ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference](https://mlsanity.com/paper/2312.02554/) (2023)
- [Learning Linear Utility Functions From Pairwise Comparison Queries](https://mlsanity.com/paper/2405.02612/) (2024)
- [Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier](https://mlsanity.com/paper/2405.17956/) (2024)
- [Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspective](https://mlsanity.com/paper/2501.18282/) (2025)
- [Learning Parametric Distributions from Samples and Preferences](https://mlsanity.com/paper/2505.23557/) (2025)

## Abstract (arXiv)

We study parameter estimation when each sample from a continuous parametric distribution is accompanied by a binary quality label governed by the same unknown parameter. By imposing hard feasibility constraints, deterministic labels can accelerate estimation from the classical n-1/2 rate of smooth quality-aware M-estimators to n-1. Combining PAC guarantees with local error-to-parameter conversions, we establish this rate for both identifiable and finitely ambiguous quality classifiers, with likelihood information resolving the latter ambiguity. When the quality model is known only up to an additive constant, we construct threshold-free pairwise preferences from oppositely labeled samples and retain the accelerated rate despite the unknown threshold. We further show that pointwise qualities and pairwise preferences are incomparable: either can be strictly more informative than the other. Our assumptions hold for Gaussian, Laplace, and exponential families under thresholded log-likelihood rewards. Finally, a minimax lower bound establishes the optimal n-1 dependence on sample size, up to logarithmic and dimension-dependent factors, while experiments confirm the benefits of constraint-based estimation.