# Moral Machine or Tyranny of the Majority?

arXiv: https://arxiv.org/abs/2305.17319 · HTML: https://mlsanity.com/paper/2305.17319/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 36 references that are in the map.
- First posted on arXiv 27 May 2023.

## Builds on (most cited first, 7 of 7)

- [A Voting-Based System for Ethical Decision Making](https://mlsanity.com/paper/1709.06692/) (2017)
- [Metric Learning for Individual Fairness](https://mlsanity.com/paper/1906.00250/) (2019)
- [Adapting a Kidney Exchange Algorithm to Align with Human Values](https://mlsanity.com/paper/2005.09755/) (2020)
- [Fair Performance Metric Elicitation](https://mlsanity.com/paper/2006.12732/) (2020)
- [On the Strategyproofness of the Geometric Median](https://mlsanity.com/paper/2106.02394/) (2021)
- [Quadratic Metric Elicitation for Fairness and Beyond](https://mlsanity.com/paper/2011.01516/) (2020)
- [Privacy-preserving Crowd-guided AI Decision-making in Ethical Dilemmas](https://mlsanity.com/paper/1906.01562/) (2019)

## Built on by (most cited first, 22 of 22)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards](https://mlsanity.com/paper/2402.18571/) (2024)
- [Personalized Language Modeling from Personalized Human Feedback](https://mlsanity.com/paper/2402.05133/) (2024)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [The History and Risks of Reinforcement Learning and Human Feedback](https://mlsanity.com/paper/2310.13595/) (2023)
- [AI Alignment at Your Discretion](https://mlsanity.com/paper/2502.10441/) (2025)
- [Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia](https://mlsanity.com/paper/2402.14147/) (2024)
- [Proportional Aggregation of Preferences for Sequential Decision Making](https://mlsanity.com/paper/2306.14858/) (2023)
- [Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges](https://mlsanity.com/paper/2308.00031/) (2023)
- [Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback](https://mlsanity.com/paper/2412.19436/) (2024)
- [Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisions](https://mlsanity.com/paper/2503.00940/) (2025)
- [Pareto-Optimal Learning from Preferences with Hidden Context](https://mlsanity.com/paper/2406.15599/) (2024)
- [Societal Alignment Frameworks Can Improve LLM Alignment](https://mlsanity.com/paper/2503.00069/) (2025)
- [On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods](https://mlsanity.com/paper/2408.02862/) (2024)
- [Benchmarking Overton Pluralism in LLMs](https://mlsanity.com/paper/2512.01351/) (2025)
- [Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2603.12595/) (2026)
- [Influencing Humans to Conform to Preference Models for RLHF](https://mlsanity.com/paper/2501.06416/) (2025)
- [Intuitions of Compromise: Utilitarianism vs. Contractualism](https://mlsanity.com/paper/2410.05496/) (2024)
- [The End Justifies the Mean: A Linear Ranking Rule for Proportional Sequential Decisions](https://mlsanity.com/paper/2605.12717/) (2026)
- [Exploring Moral Exercises for Human Oversight of AI systems: Insights from Three Pilot Studies](https://mlsanity.com/paper/2505.15851/) (2025)
- [Cautious Context Steering for Language Model Personalization](https://mlsanity.com/paper/2608.05813/) (2026)

## Abstract (arXiv)

With Artificial Intelligence systems increasingly applied in consequential domains, researchers have begun to ask how these systems ought to act in ethically charged situations where even humans lack consensus. In the Moral Machine project, researchers crowdsourced answers to "Trolley Problems" concerning autonomous vehicles. Subsequently, Noothigattu et al. (2018) proposed inferring linear functions that approximate each individual's preferences and aggregating these linear models by averaging parameters across the population. In this paper, we examine this averaging mechanism, focusing on fairness concerns in the presence of strategic effects. We investigate a simple setting where the population consists of two groups, with the minority constituting an α < 0.5 share of the population. To simplify the analysis, we consider the extreme case in which within-group preferences are homogeneous. Focusing on the fraction of contested cases where the minority group prevails, we make the following observations: (a) even when all parties report their preferences truthfully, the fraction of disputes where the minority prevails is less than proportionate in α; (b) the degree of sub-proportionality grows more severe as the level of disagreement between the groups increases; (c) when parties report preferences strategically, pure strategy equilibria do not always exist; and (d) whenever a pure strategy equilibrium exists, the majority group prevails 100% of the time. These findings raise concerns about stability and fairness of preference vector averaging as a mechanism for aggregating diverging voices. Finally, we discuss alternatives, including randomized dictatorship and median-based mechanisms.