# Evaluation Framework for AI Systems in "the Wild"

arXiv: https://arxiv.org/abs/2504.16778 · HTML: https://mlsanity.com/paper/2504.16778/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2025.

## Built on by (most cited first, 4 of 4)

- [OneMillion-Bench: How Far are Language Agents from Human Experts?](https://mlsanity.com/paper/2603.07980/) (2026)
- [Investigating Political and Demographic Associations in Large Language Models Through Moral Foundations Theory](https://mlsanity.com/paper/2510.13902/) (2025)
- [Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research](https://mlsanity.com/paper/2604.17843/) (2026)
- [Will Power Return to the Clouds? From Divine Authority to GenAI Authority](https://mlsanity.com/paper/2512.03076/) (2025)

## Abstract (arXiv)

Generative AI (GenAI) models have become vital across industries, yet current evaluation methods have not adapted to their widespread use. Traditional evaluations often rely on benchmarks and fixed datasets, frequently failing to reflect real-world performance, which creates a gap between lab-tested outcomes and practical applications. This white paper proposes a comprehensive framework for how we should evaluate real-world GenAI systems, emphasizing diverse, evolving inputs and holistic, dynamic, and ongoing assessment approaches. The paper offers guidance for practitioners on how to design evaluation methods that accurately reflect real-time capabilities, and provides policymakers with recommendations for crafting GenAI policies focused on societal impacts, rather than fixed performance numbers or parameter sizes. We advocate for holistic frameworks that integrate performance, fairness, and ethics and the use of continuous, outcome-oriented methods that combine human and automated assessments while also being transparent to foster trust among stakeholders. Implementing these strategies ensures GenAI models are not only technically proficient but also ethically responsible and impactful.