# Multiple latent orderings better predict language model preferences

arXiv: https://arxiv.org/abs/2609.22170 · HTML: https://mlsanity.com/paper/2609.22170/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 38 of its 66 references that are in the map.
- First posted on arXiv 26 Aug 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 38)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF](https://mlsanity.com/paper/2312.08358/) (2023)
- [Questioning the Survey Responses of Large Language Models](https://mlsanity.com/paper/2306.07951/) (2023)
- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [Are Large Language Models Consistent over Value-laden Questions?](https://mlsanity.com/paper/2407.02996/) (2024)
- [PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences](https://mlsanity.com/paper/2406.08469/) (2024)
- [Investigating Non-Transitivity in LLM-as-a-Judge](https://mlsanity.com/paper/2502.14074/) (2025)
- [Efficient computation of rankings from pairwise comparisons](https://mlsanity.com/paper/2207.00076/) (2022)
- [BlackJAX: Composable Bayesian inference in JAX](https://mlsanity.com/paper/2402.10797/) (2024)
- [Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments](https://mlsanity.com/paper/2406.11370/) (2024)
- [Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences](https://mlsanity.com/paper/2405.15065/) (2024)
- [TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them](https://mlsanity.com/paper/2509.21117/) (2025)
- [Direct Alignment with Heterogeneous Preferences](https://mlsanity.com/paper/2502.16320/) (2025)
- [LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking](https://mlsanity.com/paper/2406.00231/) (2024)
- [MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning](https://mlsanity.com/paper/2505.24846/) (2025)
- [Identifying Multiple Personalities in Large Language Models with External Evaluation](https://mlsanity.com/paper/2402.14805/) (2024)
- [The Fragility Of Moral Judgment In Large Language Models](https://mlsanity.com/paper/2603.05651/) (2026)
- [A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth](https://mlsanity.com/paper/2601.21817/) (2026)

## Abstract (arXiv)

Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item A to B and B to C, while also preferring C to A. Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.