mlsanityData to 7 Oct 2026

Multiple latent orderings better predict language model preferences

arXiv 2609.22170 ↗First posted 26 Aug 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item A to B and B to C, while also preferring C to A. Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 66 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

38 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026How AI Systems Think About Education: Analyzing Latent Preference Patterns in Large Language Models28 Feb
  2. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments17 Jun
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference7 Mar
  4. 2023Can LLMs Capture Human Preferences?4 May

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  6. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  7. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  8. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  9. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  10. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  11. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  12. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
22 more
  1. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  2. 2023Questioning the Survey Responses of Large Language Modelscited by 70
  3. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
  4. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  5. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  6. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  7. 2022Efficient computation of rankings from pairwise comparisonscited by 31
  8. 2024BlackJAX: Composable Bayesian inference in JAXcited by 24
  9. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  10. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  11. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  12. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  13. 2024Identifying Multiple Personalities in Large Language Models with External Evaluationcited by 10
  14. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 9
  15. 2026The Fragility Of Moral Judgment In Large Language Modelscited by 8
  16. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truthcited by 7
  17. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  18. 2023Luck, skill, and depth of competition in games and social hierarchiescited by 5
  19. 2025ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstructioncited by 5
  20. 2026Scaling Laws for Moral Machine Judgment in Large Language Modelscited by 3
  21. 2023Modeling Preferences: A Bayesian Mixture of Finite Mixtures for Rankings and Ratingscited by 2
  22. 2024Modeling cyclicality and intransitivity in paired comparisons datacited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?
  2. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models
  3. 2026Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models
  4. 2026When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
  5. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
  6. 2026Incoherent Values? Probing LLM Preferences Through Parametric Variation
  7. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
  8. 2026Evaluating LLM-Generated Preference Distributions