# Beyond Fairness Metrics: Roadblocks and Challenges for Ethical AI in Practice

arXiv: https://arxiv.org/abs/2108.06217 · HTML: https://mlsanity.com/paper/2108.06217/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 85 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 22 of 22)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)
- [Principles alone cannot guarantee ethical AI](https://mlsanity.com/paper/1906.06668/) (2019)
- [The Ethics of AI Ethics -- An Evaluation of Guidelines](https://mlsanity.com/paper/1903.03425/) (2019)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Estimating Example Difficulty Using Variance of Gradients](https://mlsanity.com/paper/2008.11600/) (2020)
- [Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination](https://mlsanity.com/paper/1906.00285/) (2019)
- [Decisions, Counterfactual Explanations and Strategic Behavior](https://mlsanity.com/paper/2002.04333/) (2020)
- [The Grey Hoodie Project: Big Tobacco, Big Tech, and the threat on academic integrity](https://mlsanity.com/paper/2009.13676/) (2020)
- [Evaluating Fairness of Machine Learning Models Under Uncertain and Incomplete Information](https://mlsanity.com/paper/2102.08410/) (2021)
- [Regulatory Markets for AI Safety](https://mlsanity.com/paper/2001.00078/) (2019)
- [Debiasing classifiers: is reality at variance with expectation?](https://mlsanity.com/paper/2011.02407/) (2020)
- [Towards Self-Regulating AI: Challenges and Opportunities of AI Model Governance in Financial Services](https://mlsanity.com/paper/2010.04827/) (2020)
- [Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society](https://mlsanity.com/paper/2001.04335/) (2020)
- [Towards organizational guidelines for the responsible use of AI](https://mlsanity.com/paper/2001.09758/) (2020)

## Built on by (most cited first, 4 of 4)

- [Users are the North Star for AI Transparency](https://mlsanity.com/paper/2303.05500/) (2023)
- [Beyond Mahalanobis-Based Scores for Textual OOD Detection](https://mlsanity.com/paper/2211.13527/) (2022)
- [Talking About the Assumption in the Room](https://mlsanity.com/paper/2502.13268/) (2025)
- [Artificial Intelligence and Diabetes Mellitus: An Inside Look Through the Retina](https://mlsanity.com/paper/2402.18600/) (2024)

## Abstract (arXiv)

We review practical challenges in building and deploying ethical AI at the scale of contemporary industrial and societal uses. Apart from the purely technical concerns that are the usual focus of academic research, the operational challenges of inconsistent regulatory pressures, conflicting business goals, data quality issues, development processes, systems integration practices, and the scale of deployment all conspire to create new ethical risks. Such ethical concerns arising from these practical considerations are not adequately addressed by existing research results. We argue that a holistic consideration of ethics in the development and deployment of AI systems is necessary for building ethical AI in practice, and exhort researchers to consider the full operational contexts of AI systems when assessing ethical risks.