Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment
On the map: LLM alignment / RLHF / preference optimisation → Medical LLM alignment
Abstract (arXiv)
We introduce a method to measure the alignment between public will and language model (LM) behavior that can be applied to fine-tuning, online oversight, and pre-release safety checks. Our `chain of alignment' (CoA) approach produces a rule based reward (RBR) by creating model behavior textitrules aligned to normative textitobjectives aligned to textitpublic will. This factoring enables a nonexpert public to directly specify their will through the normative objectives, while expert intelligence is used to figure out rules entailing model behavior that best achieves those objectives. We validate our approach by applying it across three different domains of LM prompts related to mental health. We demonstrate a public input process built on collective dialogues and bridging-based ranking that reliably produces normative objectives supported by at least 96% ± 2% of the US public. We then show that rules developed by mental health experts to achieve those objectives enable a RBR that evaluates an LM response's alignment with the objectives similarly to human experts (Pearson's r=0.841, AUC=0.964). By measuring alignment with objectives that have near unanimous public support, these CoA RBRs provide an approximate measure of alignment between LM behavior and public will.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 21 references that are in the map. First posted on arXiv 15 Nov 2024.
Builds on
16 of 21 references.
References · 16
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2020Learning to summarize from human feedbackcited by 2,588
- 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
- 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
- 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
- 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 123
- 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
- 2024Rule Based Rewards for Language Model Safetycited by 108
- 2023Specific versus General Principles for Constitutional AIcited by 47
4 more
- 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 41
- 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
- 2023Democratic Policy Development using Collective Dialogues and AIcited by 17
- 2021Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reductioncited by 9
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Alignment Plausibility: A New Standard for Assuring AI in Healthcare
- 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
- 2026From Constitutions to Control: Interpretable Rewards for Aligning Language Models
- 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
- 2025Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment
- 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
- 2025Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge