# Preference Reasoning under Indeterminacy in Large Language Models

arXiv: https://arxiv.org/abs/2608.18631 · HTML: https://mlsanity.com/paper/2608.18631/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 26 of its 83 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 26 of 26)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity](https://mlsanity.com/paper/2506.06941/) (2025)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [Game-theoretic LLM: Agent Workflow for Negotiation Games](https://mlsanity.com/paper/2411.05990/) (2024)
- [Bayesian Preference Elicitation with Language Models](https://mlsanity.com/paper/2403.05534/) (2024)
- [LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory](https://mlsanity.com/paper/2502.20432/) (2025)
- [Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamics](https://mlsanity.com/paper/2512.07462/) (2025)
- [Top-K Ranking from Pairwise Comparisons: When Spectral Ranking is Optimal](https://mlsanity.com/paper/1603.04153/) (2016)
- [EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agents](https://mlsanity.com/paper/2503.18825/) (2025)
- [Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Units](https://mlsanity.com/paper/2601.21996/) (2026)
- [When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs](https://mlsanity.com/paper/2602.06286/) (2026)
- [Accelerated Preference Elicitation with LLM-Based Proxies](https://mlsanity.com/paper/2501.14625/) (2025)
- [Asking Clarifying Questions for Preference Elicitation With Large Language Models](https://mlsanity.com/paper/2510.12015/) (2025)
- [The Illusion of Rationality: Tacit Bias and Strategic Dominance in Frontier LLM Negotiation Games](https://mlsanity.com/paper/2512.09254/) (2025)
- [Discovering Differences in Strategic Behavior Between Humans and LLMs](https://mlsanity.com/paper/2602.10324/) (2026)
- [Social Welfare Function Leaderboard: When LLM Agents Allocate Social Welfare](https://mlsanity.com/paper/2510.01164/) (2025)
- [Balanced House Allocation](https://mlsanity.com/paper/2109.01992/) (2021)
- [Large Language Models for Designing Participatory Budgeting Rules](https://mlsanity.com/paper/2602.09349/) (2026)
- [Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs](https://mlsanity.com/paper/2605.07806/) (2026)

## Abstract (arXiv)

As large language models evolve into decision-making agents, the ability to reason over preferences becomes fundamental to alignment, coordination, and collective intelligence. Yet, unlike standard benchmarks, real-world preference reasoning is inherently indeterminate: information may be incomplete, and valid solutions may not exist. We argue that indeterminacy, rather than correctness alone, is a central challenge for AI reasoning. We formalize this challenge along two axes, (i) epistemic indeterminacy, arising from incomplete, partial, or expressive preferences, and (ii) structural indeterminacy, arising from the non-existence of solutions under standard social choice concepts. Across a hierarchy of tasks, we show that state-of-the-art language models systematically fail to distinguish between determined and undetermined instances, exhibiting miscalibrated reasoning even in verification settings.