Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
As Large Language Models (LLMs) become more powerful and autonomous, they increasingly face conflicts and dilemmas in many scenarios. We first summarize and taxonomize these diverse conflicts. Then, we model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution. This graph reveals that a unified stable LLM alignment is very challenging, because the graph is neither static nor necessarily consistent in different contexts. Besides, it also reveals a potential vulnerability: priority hacking, where adversaries can craft deceptive contexts to manipulate the graph and bypass safety alignments. To counter this, we propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation. While this approach enhances robustness, we also acknowledge that many ethical and value dilemmas are philosophically irreducible, posing a long-term, open challenge for the future of AI alignment.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 47 references that are in the map. First posted on arXiv 16 Mar 2026.
Builds on
38 of 47 references.
Closest ancestors · 8
Same direction, newest first
- 2025Internal Value Alignment in Large Language Models through Controlled Value Vector Activation15 Jul
- 2024Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective31 Dec
- 2024Can Language Models Reason about Individualistic Human Values and Preferences?4 Oct
- 2024Aligning LLMs with Individual Preferences via Interaction4 Oct
- 2024Personality Alignment of Large Language Models21 Aug
- 2024Language Model Alignment in Multilingual Trolley Problems2 Jul
- 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration22 Jun
- 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
Other references · 30
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
- 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
- 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
18 more
- 2024Knowledge Conflicts for LLMs: A Surveycited by 274
- 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
- 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
- 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
- 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
- 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 122
- 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
- 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
- 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
- 2024Do LLMs write like humans? Variation in grammatical and rhetorical stylescited by 62
- 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 57
- 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
- 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 47
- 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
- 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
- 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
- 2024Do LLMs have Consistent Values?cited by 13
- 2025Are Language Models Consequentialist or Deontological Moral Reasoners?cited by 12
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
- 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
- 2023Large Language Model Alignment: A Survey
- 2025Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimization
- 2023The Alignment Problem in Context
- 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only
- 2025Evaluating AI Alignment in LLMs: Output Analysis of Value Priorities Across 75 Models with Human Benchmarking
- 2026ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment