# Hard Choices in Artificial Intelligence

arXiv: https://arxiv.org/abs/2106.11022 · HTML: https://mlsanity.com/paper/2106.11022/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 140 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Fairness in Machine Learning: Lessons from Political Philosophy](https://mlsanity.com/paper/1712.03586/) (2017)
- [Principles alone cannot guarantee ethical AI](https://mlsanity.com/paper/1906.06668/) (2019)
- [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018)
- [Meta-Inverse Reinforcement Learning with Probabilistic Context Variables](https://mlsanity.com/paper/1909.09314/) (2019)
- [Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)](https://mlsanity.com/paper/1901.00064/) (2018)
- [On the Utility of Model Learning in HRI](https://mlsanity.com/paper/1901.01291/) (2019)
- [Should Robots be Obedient?](https://mlsanity.com/paper/1705.09990/) (2017)
- [A Broader View on Bias in Automated Decision-Making: Reflecting on Epistemology and Dynamics](https://mlsanity.com/paper/1807.00553/) (2018)
- [Regulating Artificial Intelligence: Proposal for a Global Solution](https://mlsanity.com/paper/2005.11072/) (2020)
- [Privacy Engineering Meets Software Engineering. On the Challenges of Engineering Privacy ByDesign](https://mlsanity.com/paper/2007.08613/) (2020)

## Built on by (most cited first, 26 of 26)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Explore, Establish, Exploit: Red Teaming Language Models from Scratch](https://mlsanity.com/paper/2306.09442/) (2023)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [The History and Risks of Reinforcement Learning and Human Feedback](https://mlsanity.com/paper/2310.13595/) (2023)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [System Safety and Artificial Intelligence](https://mlsanity.com/paper/2202.09292/) (2022)
- [Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows](https://mlsanity.com/paper/2201.07677/) (2022)
- [ACROCPoLis: A Descriptive Framework for Making Sense of Fairness](https://mlsanity.com/paper/2304.11217/) (2023)
- [Understanding AI Trustworthiness: A Scoping Review of AIES & FAccT Articles](https://mlsanity.com/paper/2510.21293/) (2025)
- [Regulation and NLP (RegNLP): Taming Large Language Models](https://mlsanity.com/paper/2310.05553/) (2023)
- [Optimization's Neglected Normative Commitments](https://mlsanity.com/paper/2305.17465/) (2023)
- [Closed-Loop View of the Regulation of AI: Equal Impact across Repeated Interactions](https://mlsanity.com/paper/2209.01410/) (2022)
- [The Value-Sensitive Conversational Agent Co-Design Framework](https://mlsanity.com/paper/2310.11848/) (2023)
- [Developing and Evaluating a Design Method for Positive Artificial Intelligence](https://mlsanity.com/paper/2402.01499/) (2024)
- [Explainable AI Systems Must Be Contestable: Here's How to Make It Happen](https://mlsanity.com/paper/2506.01662/) (2025)
- [CIRCLE: A Framework for Evaluating AI from a Real-World Lens](https://mlsanity.com/paper/2602.24055/) (2026)
- [Positive AI: Key Challenges in Designing Artificial Intelligence for Wellbeing](https://mlsanity.com/paper/2304.12241/) (2023)
- [Demanding and Designing Aligned Cognitive Architectures](https://mlsanity.com/paper/2112.10190/) (2021)
- [From Data-Driven to Purpose-Driven Artificial Intelligence: Systems Thinking for Data-Analytic Automation of Patient Care](https://mlsanity.com/paper/2506.13584/) (2025)
- [Normative Moral Pluralism for AI: A Framework for Deliberation in Complex Moral Contexts](https://mlsanity.com/paper/2508.08333/) (2025)
- [From Mimicry to True Intelligence (TI) -- A New Paradigm for Artificial General Intelligence](https://mlsanity.com/paper/2509.14474/) (2025)
- [RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies](https://mlsanity.com/paper/2603.11001/) (2026)
- [Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemma](https://mlsanity.com/paper/2603.13294/) (2026)
- [Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment](https://mlsanity.com/paper/2609.05036/) (2026)

## Abstract (arXiv)

As AI systems are integrated into high stakes social domains, researchers now examine how to design and operate them in a safe and ethical manner. However, the criteria for identifying and diagnosing safety risks in complex social contexts remain unclear and contested. In this paper, we examine the vagueness in debates about the safety and ethical behavior of AI systems. We show how this vagueness cannot be resolved through mathematical formalism alone, instead requiring deliberation about the politics of development as well as the context of deployment. Drawing from a new sociotechnical lexicon, we redefine vagueness in terms of distinct design challenges at key stages in AI system development. The resulting framework of Hard Choices in Artificial Intelligence (HCAI) empowers developers by 1) identifying points of overlap between design decisions and major sociotechnical challenges; 2) motivating the creation of stakeholder feedback channels so that safety issues can be exhaustively addressed. As such, HCAI contributes to a timely debate about the status of AI development in democratic societies, arguing that deliberation should be the goal of AI Safety, not just the procedure by which it is ensured.