# CIRCLE: A Framework for Evaluating AI from a Real-World Lens

arXiv: https://arxiv.org/abs/2602.24055 · HTML: https://mlsanity.com/paper/2602.24055/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 76 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Builds on (most cited first, 25 of 25)

- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice](https://mlsanity.com/paper/2310.00907/) (2023)
- [Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge](https://mlsanity.com/paper/2502.00561/) (2025)
- [Measurement to Meaning: A Validity-Centered Framework for AI Evaluation](https://mlsanity.com/paper/2505.10573/) (2025)
- [Rethinking Model Evaluation as Narrowing the Socio-Technical Gap](https://mlsanity.com/paper/2306.03100/) (2023)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [Hard Choices in Artificial Intelligence](https://mlsanity.com/paper/2106.11022/) (2021)
- [Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction](https://mlsanity.com/paper/2409.16783/) (2024)
- [Operationalizing the Legal Principle of Data Minimization for Personalization](https://mlsanity.com/paper/2005.13718/) (2020)
- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impacts](https://mlsanity.com/paper/2412.01934/) (2024)
- [Understanding Teen Overreliance on AI Companion Chatbots Through Self-Reported Reddit Narratives](https://mlsanity.com/paper/2507.15783/) (2025)
- [Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes](https://mlsanity.com/paper/2307.05862/) (2023)
- [Simulating Psychological Risks in Human-AI Interactions: Real-Case Informed Modeling of AI-Induced Addiction, Anorexia, Depression, Homicide, Psychosis, and Suicide](https://mlsanity.com/paper/2511.08880/) (2025)
- [From human-centered to social-centered artificial intelligence: Assessing ChatGPT's impact through disruptive events](https://mlsanity.com/paper/2306.00227/) (2023)
- [Validity in Design Science](https://mlsanity.com/paper/2503.09466/) (2025)
- [Attention to Non-Adopters](https://mlsanity.com/paper/2510.15951/) (2025)
- [Exploring How LLMs Capture and Represent Domain-Specific Knowledge](https://mlsanity.com/paper/2504.16871/) (2025)
- [Secondary Stakeholders in AI: Fighting for, Brokering, and Navigating Agency](https://mlsanity.com/paper/2506.07281/) (2025)
- [A Different Approach to AI Safety: Proceedings from the Columbia Convening on Openness in Artificial Intelligence and AI Safety](https://mlsanity.com/paper/2506.22183/) (2025)
- [Limits of Safe AI Deployment: Differentiating Oversight and Control](https://mlsanity.com/paper/2507.03525/) (2025)
- [Red Teaming AI Red Teaming](https://mlsanity.com/paper/2507.05538/) (2025)
- [Ask What Your Country Can Do For You: Towards a Public Red Teaming Model](https://mlsanity.com/paper/2510.20061/) (2025)
- [The AI Invisibility Effect: Understanding Human-AI Interaction When Users Don't Recognize Artificial Intelligence](https://mlsanity.com/paper/2601.00579/) (2026)

## Built on by (most cited first, 2 of 2)

- [ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents](https://mlsanity.com/paper/2605.20251/) (2026)
- [How Far Can Machine Translation Quality Take You? Extrinsic Discourse Evaluation in Goal-Oriented Setups](https://mlsanity.com/paper/2606.16596/) (2026)

## Abstract (arXiv)

This study proposes CIRCLE, a six-stage, lifecycle-based framework to bridge the reality gap between model-centric performance metrics and AI system outcomes in deployment. Current approaches such as MLOps frameworks and AI model benchmarks offer detailed insights into system stability and model capabilities, but they do not provide decision makers outside the AI stack with systematic evidence of how these systems actually behave in real world contexts or affect their organizations over time. CIRCLE operationalizes the Validation phase of TEVV (Test, Evaluation, Verification, and Validation) by translating priorities of stakeholders outside the stack into measurable signals. Unlike participatory design which often remains localized, or algorithmic audits which are often retrospective, CIRCLE provides a structured, prospective protocol for linking context sensitive qualitative insights to scalable quantitative metrics. By integrating methods such as field testing, red teaming, and longitudinal studies into a coordinated pipeline, CIRCLE produces systematic knowledge; evidence that is comparable across sites yet sensitive to local context. This can enable governance based on materialized downstream effects rather than theoretical capabilities.