mlsanityData to 14 Oct 2026

Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model

arXiv 2512.21917 ↗First posted 26 Dec 2025cs.LG cs.AI econ.EMcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Policy alignment to preference data typically assumes a known link function between observed preferences and latent rewards (e.g., Bradley-Terry model / logistic link). Misspecification of this link can bias inferred rewards and misalign learned policies. We study policy alignment under an unknown and unrestricted link function. We formulate an f-divergence-constrained reward maximization problem and show that realizability in a policy class induces a semiparametric single-index binary choice model, where a scalar policy-induced index captures all dependence on demonstrations and the remaining preference distribution is unrestricted. Rather than impose identifiability of structural parameters of such a model and estimate them, as in econometrics, we develop methods that directly learn policies, with the reward function implicit, analyzing error to the optimal policy and allowing for unidentifiable and nonparametric indices. We prove link-agnostic convergence guarantees in terms of generic function complexity measures and validate the methods and theory empirically. Code is available at https://github.com/causalml/spo/.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 84 references that are in the map. First posted on arXiv 26 Dec 2025.

Builds on

23 of 84 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences17 Oct
  2. 2025Doubly Robust Alignment for Large Language Models1 Jun
  3. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?29 May
  4. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  5. 2024ROPO: Robust Preference Optimization for Large Language Models5 Apr
  6. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep

Other references · 16

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  9. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  10. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  11. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  12. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
4 more
  1. 2019Localized Debiased Machine Learning: Efficient Inference on Quantile Treatment Effects and Beyondcited by 23
  2. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  3. 2024Strong Preferences Affect the Robustness of Preference Models and Value Alignmentcited by 4
  4. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Causal methods for LLM development and evaluationcited by 2
  2. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  2. 2024Robust Preference Optimization through Reward Model Distillation
  3. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework
  4. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  5. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory
  6. 2025Direct Alignment with Heterogeneous Preferences
  7. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  8. 2025Why DPO is a Misspecified Estimator and How to Fix It