mlsanityData to 7 Oct 2026

Beyond Aggregate Scores: Behavioral Correctness Assumptions for Assessing Reference-Based Automatic Evaluation Methods

arXiv 2609.05289 ↗First posted 4 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Automated reference-based evaluation methods play a critical role in assessing natural language generation systems. Existing meta-evaluation primarily measures agreement with human judgments or benchmark labels, providing limited insight into evaluator behavior under controlled conditions. We introduce behavioral correctness assumptions, a complementary framework for evaluating reference-based automatic evaluation methods. We define a taxonomy of correctness-preserving and correctness-altering assumptions and operationalize them through controlled response transformations that specify expected scoring behaviors. We evaluate diverse lexical, character-level, semantic, LLM-based, and hybrid evaluators and analyze their assumption-level behavior, stability, sensitivity, repeat-run variability, configuration sensitivity, and reproducibility. Our experiments reveal distinct behavioral trade-offs across evaluation paradigms: no evaluator satisfies all proposed correctness assumptions, and evaluators with similar aggregate performance can exhibit substantially different behavioral profiles. These findings demonstrate that behavioral correctness assumptions provide diagnostic information obscured by conventional aggregate meta-evaluation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

22 of 26 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  11. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  12. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
10 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  4. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  6. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  7. 2024A Survey of Useful LLM Evaluationcited by 28
  8. 2024Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutionscited by 17
  9. 2024RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systemscited by 4
  10. 2026Semantic Needles in Document Haystacks: Sensitivity Testing of LLM-as-a-Judge Similarity Scoringcited by 2

Built on by

Not cited yet.