Why Expert Alignment Is Hard: Evidence from Subjective Evaluation
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
Aligning large language models with expert judgment is especially difficult in subjective evaluation tasks, where experts may disagree, rely on tacit criteria, and change their judgments over time. In this paper, we study expert alignment as a way to understand this difficulty. Using expert evaluations and follow-up questionnaires, we examine how different forms of expert information affect alignment and what this reveals about subjective judgment. Our findings show four consistent patterns. First, alignment difficulty varies substantially across experts, suggesting that expert evaluation styles differ widely in their distance from a model's prior behavior. Second, explicit criteria and reasoning do not always improve alignment, indicating that expert judgment is not fully captured by verbalized rules. Third, editing is sensitive to both the number and the identity of examples, with small numbers of edits providing useful but unstable gains. Fourth, alignment difficulty differs across evaluation dimensions: dimensions grounded more directly in proposal content are easier to align, while dimensions requiring external knowledge or value-based judgment remain harder. Taken together, these results suggest that expert alignment is difficult not only because of model limitations, but also because subjective evaluation is inherently heterogeneous, partly tacit, dimension-dependent, and temporally unstable.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 17 references that are in the map. First posted on arXiv 6 May 2026.
Builds on
14 of 17 references.
Closest ancestors · 2
Same direction, newest first
- 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment17 Apr
- 2024PAD: Personalized Alignment of LLMs at Decoding-Time5 Oct
Other references · 12
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2022Mass-Editing Memory in a Transformercited by 1,001
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
- 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
- 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 112
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models
- 2026Pressure Reveals Character: Behavioural Alignment Evaluation at Depth
- 2024Revisiting the Superficial Alignment Hypothesis
- 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
- 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability
- 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
- 2024Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation
- 2026How AI Systems Think About Education: Analyzing Latent Preference Patterns in Large Language Models