# Generative-Evaluative Agreement: A Necessary Validity Criterion for LLM-Enabled Adaptive Assessment

arXiv: https://arxiv.org/abs/2605.19529 · HTML: https://mlsanity.com/paper/2605.19529/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 11 of 11)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- [Evaluating the Consistency of LLM Evaluators](https://mlsanity.com/paper/2412.00543/) (2024)
- [The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate](https://mlsanity.com/paper/2402.06204/) (2024)
- [Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring](https://mlsanity.com/paper/2407.18328/) (2024)
- [AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition](https://mlsanity.com/paper/2509.21910/) (2025)
- [Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AI](https://mlsanity.com/paper/2511.06078/) (2025)

## Built on by (most cited first, 1 of 1)

- [Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study](https://mlsanity.com/paper/2609.32638/) (2026)

## Abstract (arXiv)

When the same LLM generates assessment items, simulates student responses, and scores them, the validation loop is self-referential. We introduce Generative-Evaluative Agreement (GEA), a validity criterion measuring whether an LLM's scoring function recovers the skill levels its generative function was instructed to produce. In the first direct measurement of GEA on a two-stage adaptive assessment, the model recovers roughly half the intended variance r = 0.698 with systematic positive bias. GEA is strong r > 0.7 for syntactically verifiable skills but near zero for design-level skills, and low-skill overestimation inflates scores near the routing threshold. We argue that granular, skill-decomposed rubrics are the principal proposed mechanism for strengthening GEA and outline complementary mitigations.