# As an AI Language Model, "Yes I Would Recommend Calling the Police": Norm Inconsistency in LLM Decision-Making

arXiv: https://arxiv.org/abs/2405.14812 · HTML: https://mlsanity.com/paper/2405.14812/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2024.

## Built on by (most cited first, 10 of 10)

- [Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs](https://mlsanity.com/paper/2506.13082/) (2025)
- [Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications](https://mlsanity.com/paper/2511.19933/) (2025)
- [Teaching AI to Handle Exceptions: Supervised Fine-Tuning with Human-Aligned Judgment](https://mlsanity.com/paper/2503.02976/) (2025)
- [Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks](https://mlsanity.com/paper/2510.03384/) (2025)
- [Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios](https://mlsanity.com/paper/2601.03553/) (2026)
- [Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals](https://mlsanity.com/paper/2602.08882/) (2026)
- [The Biased Samaritan: LLM biases in Perceived Kindness](https://mlsanity.com/paper/2506.11361/) (2025)
- [Compounding Disadvantage: Auditing Intersectional Bias in LLM-Generated Explanations Across Indian and American STEM Education](https://mlsanity.com/paper/2601.14506/) (2026)
- [Auditing demographic bias in AI-based emergency police dispatch: a cross-lingual evaluation of eleven large language models](https://mlsanity.com/paper/2605.01451/) (2026)
- [Less Is Moral: A CHARMing Framework for Moral Foundations Detection in Endorsement Behaviour](https://mlsanity.com/paper/2609.03330/) (2026)

## Abstract (arXiv)

We investigate the phenomenon of norm inconsistency: where LLMs apply different norms in similar situations. Specifically, we focus on the high-risk application of deciding whether to call the police in Amazon Ring home surveillance videos. We evaluate the decisions of three state-of-the-art LLMs -- GPT-4, Gemini 1.0, and Claude 3 Sonnet -- in relation to the activities portrayed in the videos, the subjects' skin-tone and gender, and the characteristics of the neighborhoods where the videos were recorded. Our analysis reveals significant norm inconsistencies: (1) a discordance between the recommendation to call the police and the actual presence of criminal activity, and (2) biases influenced by the racial demographics of the neighborhoods. These results highlight the arbitrariness of model decisions in the surveillance context and the limitations of current bias detection and mitigation strategies in normative decision-making.