AI Alignment Breaks at the Edge
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
General Alignment has improved average-case helpfulness and safety, but current alignment practice still rewards confident, single-turn responses. The problem is not only that models fail on edge cases; it is that current evaluation makes many of these failures hard to see. We take the position that alignment must move beyond average-case evaluation by making failures under value conflict, plural stakeholder disagreement, and epistemic ambiguity visible and actionable. Scalar rewards compress diverse values into a single number; data and evaluation regimes collapse, filter, or fail to elicit the cases where alignment is hardest; and governance often lacks mechanisms for adjudicating contested cases. These blind spots produce value flattening, representation loss, and uncertainty blindness. We use Edge alignment to name a detection, evaluation, and governance agenda for surfacing these failures and connecting them to appropriate interventions. Rather than a single training objective, Edge alignment defines the conditions under which standard alignment should yield to mechanisms that preserve multidimensional value structure, represent plural perspectives, and support uncertainty-aware interaction. A pilot diagnostic set of 91 edge cases and four contemporary models illustrates that ordinary helpfulness and safety readings can miss process failures that edge-aware evaluation exposes. We outline operational edge signals, process-aware evaluation criteria, and a three-phase process stack that reframes alignment as a lifecycle problem of dynamic normative governance.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 75 of its 112 references that are in the map. First posted on arXiv 23 Feb 2026.
Builds on
75 of 112 references.
Closest ancestors · 14
Same direction, newest first
- 2025Dropouts in Confidence: Moral Uncertainty in Human-LLM Alignment17 Nov
- 2025SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization9 Nov
- 2025Decentralising LLM Alignment: A Case for Context, Pluralism, and Participation9 Sep
- 2025Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression11 Aug
- 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications21 Mar
- 2025Value Profiles for Encoding Human Variation19 Mar
- 2024Aligning LLMs with Individual Preferences via Interaction4 Oct
- 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration22 Jun
- 2024Collective Constitutional AI: Aligning a Language Model with Public Input12 Jun
- 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
- 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
- 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback19 Sep
2 more
Other references · 61
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2021Ethical and social risks of harm from Language Modelscited by 957
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
38 more
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
- 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
- 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
- 2021Unsolved Problems in ML Safetycited by 291
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
- 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
- 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
- 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
- 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
- 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
- 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
- 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
- 2020Participation is not a Design Fix for Machine Learningcited by 110
- 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
- 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
- 2023The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practicecited by 86
- 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 78
- 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
- 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
- 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 57
- 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 48
- 2023Generative Social Choicecited by 43
- 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
- 2021Explainability Pitfalls: Beyond Dark Patterns in Explainable AIcited by 36
- 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
- 2024Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methodscited by 26
- 2023Interactive AI Alignment: Specification, Process, and Evaluation Alignmentcited by 22
- 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agentscited by 22
- 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 20
- 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
- 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
- 2024Holistic Safety and Responsibility Evaluations of Advanced AI Modelscited by 18
- 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 15
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026A Roadmap to Impactful Pluralistic Alignment Research24 Jul 2026
- 2026A Geometric Perspective on Stabilizing Value Conflict Resolution20 Jul 2026
- 2026Quantifying and Mitigating Premature Closure in Frontier LLMs14 May 2026
Similar, not cited
Close in wording, no citation either way
- 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models
- 2026Principles Do Not Apply Themselves: A Hermeneutic Perspective on AI Alignment
- 2025The Specification Trap: Why Static Value Alignment Alone Is Insufficient for Robust Alignment
- 2024Measuring Error Alignment for Decision-Making Systems
- 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
- 2026Toward a Theory of Value in AI Alignment
- 2024The AI Alignment Paradox
- 2023A Multi-Level Framework for the AI Alignment Problem