# Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms arXiv: https://arxiv.org/abs/2510.10339 ยท HTML: https://mlsanity.com/paper/2510.10339/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 80 references that are in the map. - First posted on arXiv 11 Oct 2025. ## Builds on (most cited first, 30 of 34) - [Explainability in Graph Neural Networks: A Taxonomic Survey](https://mlsanity.com/paper/2012.15445/) (2020) - [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017) - [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019) - [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018) - [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020) - [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020) - [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023) - [A Convex Framework for Fair Regression](https://mlsanity.com/paper/1706.02409/) (2017) - [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022) - [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024) - [Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support](https://mlsanity.com/paper/2112.05675/) (2021) - [Measurement to Meaning: A Validity-Centered Framework for AI Evaluation](https://mlsanity.com/paper/2505.10573/) (2025) - [Rethinking Model Evaluation as Narrowing the Socio-Technical Gap](https://mlsanity.com/paper/2306.03100/) (2023) - [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022) - [Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory](https://mlsanity.com/paper/2305.14889/) (2023) - [A survey of bias in Machine Learning through the prism of Statistical Parity for the Adult Data Set](https://mlsanity.com/paper/2003.14263/) (2020) - [Socially Aware Bias Measurements for Hindi Language Representations](https://mlsanity.com/paper/2110.07871/) (2021) - [Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling](https://mlsanity.com/paper/2402.17861/) (2024) - [Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems](https://mlsanity.com/paper/2101.09385/) (2021) - [Evaluating Generative AI Systems is a Social Science Measurement Challenge](https://mlsanity.com/paper/2411.10939/) (2024) - [Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud](https://mlsanity.com/paper/2109.03285/) (2021) - [A Framework for Fairness: A Systematic Review of Existing Fair AI Solutions](https://mlsanity.com/paper/2112.05700/) (2021) - [On Discrimination Discovery and Removal in Ranked Data using Causal Graph](https://mlsanity.com/paper/1803.01901/) (2018) - [Not one but many Tradeoffs: Privacy Vs. Utility in Differentially Private Machine Learning](https://mlsanity.com/paper/2008.08807/) (2020) - [MARS-Gym: A Gym framework to model, train, and evaluate Recommender Systems for Marketplaces](https://mlsanity.com/paper/2010.07035/) (2020) - [Beyond the ML Model: Applying Safety Engineering Frameworks to Text-to-Image Development](https://mlsanity.com/paper/2307.10312/) (2023) - [Measurement as governance in and for responsible AI](https://mlsanity.com/paper/2109.05658/) (2021) - [DP-UTIL: Comprehensive Utility Analysis of Differential Privacy in Machine Learning](https://mlsanity.com/paper/2112.12998/) (2021) - [STPA for Learning-Enabled Systems: A Survey and A New Practice](https://mlsanity.com/paper/2302.10588/) (2023) - [Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor](https://mlsanity.com/paper/2506.14652/) (2025) ## Built on by (most cited first, 5 of 5) - [From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants](https://mlsanity.com/paper/2604.05166/) (2026) - [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026) - [Exploring Human Perceptions of AI Responses: Insights from a Mixed-Methods Study on Risk Mitigation in Generative Models](https://mlsanity.com/paper/2512.01892/) (2025) - [Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI](https://mlsanity.com/paper/2607.14353/) (2026) - [A Chain Is Only as Strong as Its Weakest Link: A Scoping Review of System Integration Audits in AI](https://mlsanity.com/paper/2608.04921/) (2026) ## Abstract (arXiv) Over the past decade, an ecosystem of measures has emerged to evaluate the social and ethical implications of AI systems, largely shaped by high-level ethics principles. These measures are developed and used in fragmented ways, without adequate attention to how they are situated in AI systems. In this paper, we examine how existing measures used in the computing literature map to AI system components, attributes, hazards, and harms. Our analysis draws on a scoping review resulting in nearly 800 measures corresponding to 11 AI ethics principles. We find that most measures focus on four principles - fairness, transparency, privacy, and trust - and primarily assess model or output system components. Few measures account for interactions across system elements, and only a narrow set of hazards is typically considered for each harm type. Many measures are disconnected from where harm is experienced and lack guidance for setting meaningful thresholds. These patterns reveal how current evaluation practices remain fragmented, measuring in pieces rather than capturing how harms emerge across systems. Framing measures with respect to system attributes, hazards, and harms can strengthen regulatory oversight, support actionable practices in industry, and ground future research in systems-level understanding.