mlsanityData to 7 Oct 2026

Pairwise Reference Alignment as a Model-Level Ordinal Observable

arXiv 2605.30758 ↗First posted 29 May 2026cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Pairwise preference data is widely used in language-model evaluation and alignment, often for model ranking, reward modeling, or preference optimization. This note formulates a more basic measurement question: given a reference distribution of pairwise preferences, what model-level quantity is estimated when we test whether a model ranks preferred responses above rejected responses? We define pairwise reference alignment as an ordinal observable induced by a model scoring function. Given a reference pair distribution Ppair over triples (x,y^+,y^-), and a scalar model score SM(x,y), we define the alignment observable as the probability that the model-induced ordering agrees with the reference preference ordering. We further define a centered order-parameter-like statistic and discuss a margin-based extension. The resulting quantities admit simple finite-sample estimators and concentration bounds under independent sampling assumptions. This note does not introduce a new benchmark. It provides a conceptual and statistical formulation for pairwise reference alignment, clarifies the role of the reference pair distribution, and distinguishes the general ordinal observable from scoring choices such as normalized log-probability or energy-based scores. We also provide an initial empirical study on Qwen2.5 models and RewardBench, where the proposed statistics increase with model size and instruction tuning and vary across reference-pair subsets as predicted by the formulation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 17 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

13 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  8. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 130
  9. 2022Efficient computation of rankings from pairwise comparisonscited by 31
  10. 2023A density estimation perspective on learning from pairwise human preferencescited by 23
  11. 2023Generalized Bradley-Terry Models for Score Estimation from Paired Comparisonscited by 9
  12. 2021On Preference Learning Based on Sequential Bayesian Optimization with Pairwise Comparisoncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
  2. 2025Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback
  3. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
  4. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  5. 2025Pairwise Calibrated Rewards for Pluralistic Alignment
  6. 2025Why Alignment Must Precede Distillation: A Minimal Working Explanation
  7. 2024Permutative Preference Alignment from Listwise Ranking of Human Judgments
  8. 2025ComPO: Preference Alignment via Comparison Oracles