# Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value arXiv: https://arxiv.org/abs/2512.03399 ยท HTML: https://mlsanity.com/paper/2512.03399/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 42 of its 160 references that are in the map. - First posted on arXiv 3 Dec 2025. ## Builds on (most cited first, 30 of 42) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023) - [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017) - [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022) - [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020) - [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025) - [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024) - [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://mlsanity.com/paper/2406.10162/) (2024) - [Collective Constitutional AI: Aligning a Language Model with Public Input](https://mlsanity.com/paper/2406.07814/) (2024) - [The Ethics of Advanced AI Assistants](https://mlsanity.com/paper/2404.16244/) (2024) - [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024) - [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022) - [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018) - [Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions](https://mlsanity.com/paper/2504.15236/) (2025) - [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024) - [Why human-AI relationships need socioaffective alignment](https://mlsanity.com/paper/2502.02528/) (2025) - [Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development](https://mlsanity.com/paper/2501.16946/) (2025) - [A Mechanism-Based Approach to Mitigating Harms from Persuasive Generative AI](https://mlsanity.com/paper/2404.15058/) (2024) - [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024) - [LMUnit: Fine-grained Evaluation with Natural Language Unit Tests](https://mlsanity.com/paper/2412.13091/) (2024) - [What are human values, and how do we align AI to them?](https://mlsanity.com/paper/2404.10636/) (2024) - [Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Media](https://mlsanity.com/paper/2305.16941/) (2023) - [Characterizing AI Agents for Alignment and Governance](https://mlsanity.com/paper/2504.21848/) (2025) - [What are you optimizing for? Aligning Recommender Systems with Human Values](https://mlsanity.com/paper/2107.10939/) (2021) - [Value Profiles for Encoding Human Variation](https://mlsanity.com/paper/2503.15484/) (2025) - [A theory of appropriateness with applications to generative artificial intelligence](https://mlsanity.com/paper/2412.19010/) (2024) - [An Economy of AI Agents](https://mlsanity.com/paper/2509.01063/) (2025) - [Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AI](https://mlsanity.com/paper/2203.10525/) (2022) - [Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory](https://mlsanity.com/paper/2208.07006/) (2022) - [Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games](https://mlsanity.com/paper/2402.13399/) (2024) ## Built on by (most cited first, 6 of 6) - [Positive Alignment: Artificial Intelligence for Human Flourishing](https://mlsanity.com/paper/2605.10310/) (2026) - [Position: Capability Control Should be a Separate Goal From Alignment](https://mlsanity.com/paper/2602.05164/) (2026) - [Solipsistic Superintelligence is Unlikely to be Cooperative](https://mlsanity.com/paper/2606.03237/) (2026) - [Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction](https://mlsanity.com/paper/2607.00001/) (2026) - [A Roadmap to Impactful Pluralistic Alignment Research](https://mlsanity.com/paper/2607.22305/) (2026) - [From Certain Doom to Survival: Agent-Driven Self-Governance in LLM Agent Societies](https://mlsanity.com/paper/2609.22600/) (2026) ## Abstract (arXiv) Beneficial societal outcomes cannot be guaranteed by aligning individual AI systems with the intentions of their operators or users. Even an AI system that is perfectly aligned to the intentions of its operating organization can lead to bad outcomes if the goals of that organization are misaligned with those of other institutions and individuals. For this reason, we need full-stack alignment, the concurrent alignment of AI systems and the institutions that shape them with what people value. This can be done without imposing a particular vision of individual or collective flourishing. We argue that current approaches for representing values, such as utility functions, preference orderings, or unstructured text, struggle to address these and other issues effectively. They struggle to distinguish values from other signals, to support principled normative reasoning, and to model collective goods. We propose thick models of value will be needed. These structure the way values and norms are represented, enabling systems to distinguish enduring values from fleeting preferences, to model the social embedding of individual choices, and to reason normatively, applying values in new domains. We demonstrate this approach in five areas: AI value stewardship, normatively competent agents, win-win negotiation systems, meaning-preserving economic mechanisms, and democratic regulatory institutions.