Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt
Not on the map. Why?
Abstract (arXiv)
Artificial Intelligence (AI) systems are increasingly placed in positions where their decisions have real consequences, e.g., moderating online spaces, conducting research, and advising on policy. Ensuring they operate in a safe and ethically acceptable fashion is thus critical. However, most solutions have been a form of one-size-fits-all "alignment". We are worried that such systems, which overlook enduring moral diversity, will spark resistance, erode trust, and destabilize our institutions. This paper traces the underlying problem to an often-unstated Axiom of Rational Convergence: the idea that under ideal conditions, rational agents will converge in the limit of conversation on a single ethics. Treating that premise as both optional and doubtful, we propose what we call the appropriateness framework: an alternative approach grounded in conflict theory, cultural evolution, multi-agent systems, and institutional economics. The appropriateness framework treats persistent disagreement as the normal case and designs for it by applying four principles: (1) contextual grounding, (2) community customization, (3) continual adaptation, and (4) polycentric governance. We argue here that adopting these design principles is a good way to shift the main alignment metaphor from moral unification to a more productive metaphor of conflict management, and that taking this step is both desirable and urgent.
Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 7 of its 62 references that are in the map. First posted on arXiv 8 May 2025.
Builds on
7 of 62 references.
References · 7
- 2016Concrete Problems in AI Safetycited by 1,642
- 2019Optimal Policies Tend to Seek Powercited by 119
- 2018Incomplete Contracting and AI Alignmentcited by 62
- 2024A theory of appropriateness with applications to generative artificial intelligencecited by 25
- 2021A learning agent that acquires social norms from public sanctions in decentralized multi-agent settingscited by 24
- 2018Legible Normativity for AI Alignment: The Value of Silly Rulescited by 15
- 2020Model-free conventions in multi-agent reinforcement learning with heterogeneous preferencescited by 13
Built on by
Citing papers per year: 2025: 2; 2026: 7.
9 papers.
- 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 35
- 2025LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacracited by 22
- 2026Social physics in the age of artificial intelligencecited by 5
- 2026Positive Alignment: Artificial Intelligence for Human Flourishingcited by 5
- 2026AI as Entertainmentcited by 2
- 2026SCRuB: Social Concept Reasoning under Rubric-Based Evaluationcited by 1
- 2026Solipsistic Superintelligence is Unlikely to be Cooperativecited by 1
- 2026Distributed Legal Infrastructure for a Trustworthy Agentic Webcited by 0
- 2026From Certain Doom to Survival: Agent-Driven Self-Governance in LLM Agent Societiescited by 0
- 2026From Certain Doom to Survival: Agent-Driven Self-Governance in LLM Agent Societies18 Sep 2026
- 2026Solipsistic Superintelligence is Unlikely to be Cooperative2 Jun 2026
- 2026Positive Alignment: Artificial Intelligence for Human Flourishing11 May 2026
- 2026SCRuB: Social Concept Reasoning under Rubric-Based Evaluation7 May 2026
- 2026Distributed Legal Infrastructure for a Trustworthy Agentic Web6 Mar 2026
- 2026Social physics in the age of artificial intelligence4 Mar 2026
- 2026AI as Entertainment13 Jan 2026
- 2025LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra21 Jul 2025
- 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset13 Jul 2025