# Resource Rational Contractualism Should Guide AI Alignment

arXiv: https://arxiv.org/abs/2506.17434 · HTML: https://mlsanity.com/paper/2506.17434/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 95 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 30 of 31)

- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis](https://mlsanity.com/paper/2402.05863/) (2024)
- [PersonaGym: Evaluating Persona Agents and LLMs](https://mlsanity.com/paper/2407.18416/) (2024)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018)
- [What Would Jiminy Cricket Do? Towards Agents That Behave Morally](https://mlsanity.com/paper/2110.13136/) (2021)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [Testing Language Model Agents Safely in the Wild](https://mlsanity.com/paper/2311.10538/) (2023)
- [Infrastructure for AI Agents](https://mlsanity.com/paper/2501.10114/) (2025)
- [What are human values, and how do we align AI to them?](https://mlsanity.com/paper/2404.10636/) (2024)
- [Value Profiles for Encoding Human Variation](https://mlsanity.com/paper/2503.15484/) (2025)
- [Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning](https://mlsanity.com/paper/2402.17930/) (2024)
- [The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs](https://mlsanity.com/paper/2306.14325/) (2023)
- [Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models](https://mlsanity.com/paper/2407.08440/) (2024)
- [Reasoning About Beliefs and Actions Under Computational Resource Constraints](https://mlsanity.com/paper/1304.2759/) (2013)
- [Human Simulacra: Benchmarking the Personification of Large Language Models](https://mlsanity.com/paper/2402.18180/) (2024)
- [Goals as Reward-Producing Programs](https://mlsanity.com/paper/2405.13242/) (2024)
- [Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games](https://mlsanity.com/paper/2402.13399/) (2024)
- [AI and the Future of Digital Public Squares](https://mlsanity.com/paper/2412.09988/) (2024)
- [AGI, Governments, and Free Societies](https://mlsanity.com/paper/2503.05710/) (2025)

## Built on by (most cited first, 8 of 8)

- [Emergent alignment and the projectability of ethical personas](https://mlsanity.com/paper/2606.09475/) (2026)
- [Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction](https://mlsanity.com/paper/2607.00001/) (2026)
- [Unsupervised Elicitation of Moral Values from Language Models](https://mlsanity.com/paper/2601.17728/) (2026)
- [The Design and Composition of Structural Causal Decision Processes](https://mlsanity.com/paper/2605.02681/) (2026)
- [Bounded Morality: Defining the Space of Moral Computation](https://mlsanity.com/paper/2607.00002/) (2026)
- [A Contractualist Argumentation Framework for Moral Decision-Making](https://mlsanity.com/paper/2608.01937/) (2026)
- [Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment](https://mlsanity.com/paper/2609.05036/) (2026)
- [Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny?](https://mlsanity.com/paper/2609.05088/) (2026)

## Abstract (arXiv)

AI systems will soon have to navigate human environments and make decisions that affect people and other AI agents whose goals and values diverge. Contractualist alignment proposes grounding those decisions in agreements that diverse stakeholders would endorse under the right conditions, yet securing such agreement at scale remains costly and slow -- even for advanced AI. We therefore propose Resource-Rational Contractualism (RRC): a framework where AI systems approximate the agreements rational parties would form by drawing on a toolbox of normatively-grounded, cognitively-inspired heuristics that trade effort for accuracy. An RRC-aligned agent would not only operate efficiently, but also be equipped to dynamically adapt to and interpret the ever-changing human social world.