# Making AI Evaluation Deployment Relevant Through Context Specification

arXiv: https://arxiv.org/abs/2603.06811 · HTML: https://mlsanity.com/paper/2603.06811/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 18 of 18)

- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice](https://mlsanity.com/paper/2310.00907/) (2023)
- [Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge](https://mlsanity.com/paper/2502.00561/) (2025)
- [Measurement to Meaning: A Validity-Centered Framework for AI Evaluation](https://mlsanity.com/paper/2505.10573/) (2025)
- [Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval](https://mlsanity.com/paper/2407.20371/) (2024)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impacts](https://mlsanity.com/paper/2412.01934/) (2024)
- [Improving governance outcomes through AI documentation: Bridging theory and practice](https://mlsanity.com/paper/2409.08960/) (2024)
- [An Evaluation of Large Language Models on Text Summarization Tasks Using Prompt Engineering Techniques](https://mlsanity.com/paper/2507.05123/) (2025)
- [How Should AI Safety Benchmarks Benchmark Safety?](https://mlsanity.com/paper/2601.23112/) (2026)
- [Distinguishing Task-Specific and General-Purpose AI in Regulation](https://mlsanity.com/paper/2506.17347/) (2025)
- [On the Design and Evaluation of Human-centered Explainable AI Systems: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2510.12201/) (2025)
- [Exploring the Use of LLMs for Requirements Specification in an IT Consulting Company](https://mlsanity.com/paper/2507.19113/) (2025)
- [AI Sustainability in Practice Part One: Foundations for Sustainable AI Projects](https://mlsanity.com/paper/2403.14635/) (2024)
- [Generative AI Needs Adaptive Governance](https://mlsanity.com/paper/2406.04554/) (2024)

## Built on by (most cited first, 1 of 1)

- [VERA-MH: Validation of Ethical and Responsible AI in Mental Health](https://mlsanity.com/paper/2605.13318/) (2026)

## Abstract (arXiv)

With many organizations struggling to gain value from AI deployments, pressure to evaluate AI in an informed manner has intensified. Status quo AI evaluation approaches often mask the operational realities that ultimately determine deployment success, making it difficult for organizational decision makers to know whether and how AI tools will deliver durable value. We introduce and describe context specification as a process to support and inform this decision making process. Context specification turns diffuse stakeholder perspectives about what matters in a given setting into clear, named constructs: explicit definitions of the properties, behaviors, and outcomes that evaluations aim to capture, so they can be observed and measured in context. The process serves as a foundational roadmap for evaluating what AI systems are likely to do in the deployment contexts that organizations actually manage.