mlsanityData to 7 Oct 2026

Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean

arXiv 2604.08595 ↗First posted 4 Apr 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing evaluation methods for LLM-based AI systems, such as LLM-as-a-Judge, verdict systems, and NLI, do not always align well with human assessment because they cannot adapt their strictness to the application domain. This paper presents Temperature-Controlled Verdict Aggregation (TCVA), a method that combines a five-level verdict-scoring system with generalized power-mean aggregation and an intuitive temperature parameter T [0.1, 1.0] to control evaluation rigor. Low temperatures yield pessimistic scores suited for safety-critical domains; high temperatures produce lenient scores appropriate for conversational AI. Experimental evaluation on three benchmark datasets with human Likert-scale annotations (SummEval and USR) shows that TCVA achieves correlation with human judgments comparable to RAGAS on faithfulness (Spearman = 0.667 vs. 0.676) while consistently outperforming DeepEval. The method requires no additional LLM calls when adjusting the temperature parameter.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 21 references that are in the map. First posted on arXiv 4 Apr 2026.

Builds on

12 of 21 references.

References · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  6. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  7. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  8. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  9. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  10. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 19
  11. 2024AI Benchmarks and Datasets for LLM Evaluationcited by 7
  12. 2024An Axiomatic Approach to Loss Aggregation and an Adapted Aggregating Algorithmcited by 1

Built on by

Not cited yet.