# AI-Assisted Systematization for Evaluating GenAI Systems

arXiv: https://arxiv.org/abs/2605.26001 · HTML: https://mlsanity.com/paper/2605.26001/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 11 of 11)

- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [Taxonomizing and Measuring Representational Harms: A Look at Image Tagging](https://mlsanity.com/paper/2305.01776/) (2023)
- [Evaluating Generative AI Systems is a Social Science Measurement Challenge](https://mlsanity.com/paper/2411.10939/) (2024)
- [Fluent but Foreign: Even Regional LLMs Lack Cultural Alignment](https://mlsanity.com/paper/2505.21548/) (2025)
- [SENSE-7: Taxonomy and Dataset for Measuring User Perceptions of Empathy in Sustained Human-AI Conversations](https://mlsanity.com/paper/2509.16437/) (2025)
- [Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics](https://mlsanity.com/paper/2604.02406/) (2026)

## Built on by (most cited first, 1 of 1)

- [ASSERT: A Measurement Pipeline for GenAI Audits](https://mlsanity.com/paper/2608.13840/) (2026)

## Abstract (arXiv)

Evaluating generative AI (GenAI) systems is challenging because many targets of evaluation are broad, contested concepts, such as "reasoning," "fairness," or "creativity." When these concepts are left underspecified, it becomes unclear what should be measured or how evaluation results should be interpreted. This problem reflects a missing step: systematization, that is, moving from a broad background concept to an explicit, structured account of the concept in measurable terms. To help address the fact that systematization is cognitively demanding and resource-intensive, we investigate whether AI assistance can support this process. To enable AI-assisted systematization and assess its quality, we introduce a structured representation of a systematized concept, a concept spec, and a validation worksheet. We then develop two AI-assisted systematizers: a direct, zero-shot approach and a multi-agent approach that more closely mirrors manual systematization approaches from existing literature. We use these systematizers to produce concept specs for two concepts -- hate-based rhetoric and digital empathy -- and evaluate resulting concept specs on content validity and information recoverability.