# When Is It Acceptable to Break the Rules? Knowledge Representation of Moral Judgement Based on Empirical Data

arXiv: https://arxiv.org/abs/2201.07763 · HTML: https://mlsanity.com/paper/2201.07763/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 119 references that are in the map.
- First posted on arXiv 19 Jan 2022.

## Builds on (most cited first, 16 of 16)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Can Machines Learn Morality? The Delphi Experiment](https://mlsanity.com/paper/2110.07574/) (2021)
- [Research Priorities for Robust and Beneficial Artificial Intelligence](https://mlsanity.com/paper/1602.03506/) (2016)
- [Thinking Fast and Slow in AI](https://mlsanity.com/paper/2010.06002/) (2020)
- [A Voting-Based System for Ethical Decision Making](https://mlsanity.com/paper/1709.06692/) (2017)
- [The social dilemma of autonomous vehicles](https://mlsanity.com/paper/1510.03346/) (2015)
- [Adapting a Kidney Exchange Algorithm to Align with Human Values](https://mlsanity.com/paper/2005.09755/) (2020)
- [Building Ethically Bounded AI](https://mlsanity.com/paper/1812.03980/) (2018)
- [Building Ethics into Artificial Intelligence](https://mlsanity.com/paper/1812.02953/) (2018)
- [Incorporating Behavioral Constraints in Online AI Systems](https://mlsanity.com/paper/1809.05720/) (2018)
- [A Computational Model of Commonsense Moral Decision Making](https://mlsanity.com/paper/1801.04346/) (2018)
- [Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations](https://mlsanity.com/paper/2109.11018/) (2021)
- [The Computational Complexity of Dominance and Consistency in CP-Nets](https://mlsanity.com/paper/1401.3453/) (2014)
- [Voting with Random Classifiers (VORACE): Theoretical and Experimental Analysis](https://mlsanity.com/paper/1909.08996/) (2019)
- [Teaching Social Behavior through Human Reinforcement for Ad hoc Teamwork -The STAR Framework](https://mlsanity.com/paper/1809.07880/) (2018)

## Built on by (most cited first, 5 of 5)

- [Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents](https://mlsanity.com/paper/2404.16698/) (2024)
- [When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment](https://mlsanity.com/paper/2210.01478/) (2022)
- [ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations](https://mlsanity.com/paper/2212.10409/) (2022)
- [Morality is Contextual: Learning Interpretable Moral Contexts from Human Data with Probabilistic Clustering and Large Language Models](https://mlsanity.com/paper/2512.21439/) (2025)
- [Logic-Based Ethical Planning](https://mlsanity.com/paper/2206.00595/) (2022)

## Abstract (arXiv)

One of the most remarkable things about the human moral mind is its flexibility. We can make moral judgments about cases we have never seen before. We can decide that pre-established rules should be broken. We can invent novel rules on the fly. Capturing this flexibility is one of the central challenges in developing AI systems that can interpret and produce human-like moral judgment. This paper details the results of a study of real-world decision makers who judge whether it is acceptable to break a well-established norm: ``no cutting in line.'' We gather data on how human participants judge the acceptability of line-cutting in a range of scenarios. Then, in order to effectively embed these reasoning capabilities into a machine, we propose a method for modeling them using a preference-based structure, which captures a novel modification to standard ``dual process'' theories of moral judgment.