Principles Do Not Apply Themselves: A Hermeneutic Perspective on AI Alignment
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
AI alignment is often framed as the task of ensuring that an AI system follows a set of stated principles or human preferences, but general principles rarely determine their own application in concrete cases. When principles conflict, when they are too broad to settle a situation, or when the relevant facts are unclear, an additional act of judgment is required. This paper analyzes that step through the lens of hermeneutics and argues that alignment therefore includes an interpretive component: it involves context-sensitive judgments about how principles should be read, applied, and prioritized in practice. We connect this claim to recent empirical findings showing that a substantial portion of preference-labeling data falls into cases of principle conflict or indifference, where the principle set does not uniquely determine a decision. We then draw an operational consequence: because such judgments are expressed in behavior, many alignment-relevant choices appear only in the distribution of responses a model generates at deployment time. To formalize this point, we distinguish deployment-induced and corpus-induced evaluation and show that off-policy audits can fail to capture alignment-relevant failures when the two response distributions differ. We argue that principle-specified alignment includes a context-dependent interpretive component.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 11 of its 16 references that are in the map. First posted on arXiv 12 Apr 2026.
Builds on
11 of 16 references.
Closest ancestors · 4
Same direction, newest first
- 2024Collective Constitutional AI: Aligning a Language Model with Public Input12 Jun
- 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
- 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback16 Apr
- 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
Other references · 7
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
- 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
- 2025AI Alignment at Your Discretioncited by 20
- 2024Pluralistic Alignment Over Timecited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Beyond Preferences in AI Alignment
- 2026The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI
- 2024The AI Alignment Paradox
- 2026Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction
- 2024Measuring Error Alignment for Decision-Making Systems
- 2023A Multi-Level Framework for the AI Alignment Problem
- 2026The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers
- 2025Evaluating AI Alignment in LLMs: Output Analysis of Value Priorities Across 75 Models with Human Benchmarking