# Statistical Multicriteria Evaluation of LLM-Generated Text

arXiv: https://arxiv.org/abs/2506.18082 · HTML: https://mlsanity.com/paper/2506.18082/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 84 references that are in the map.
- First posted on arXiv 22 Jun 2025.

## Builds on (most cited first, 30 of 54)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [Benchmarking LLMs via Uncertainty Quantification](https://mlsanity.com/paper/2401.12794/) (2024)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Mapping global dynamics of benchmark creation and saturation in artificial intelligence](https://mlsanity.com/paper/2203.04592/) (2022)
- [Jointly Measuring Diversity and Quality in Text Generation Models](https://mlsanity.com/paper/1904.03971/) (2019)
- [Scientific Machine Learning Benchmarks](https://mlsanity.com/paper/2110.12773/) (2021)
- [Quantifying Variance in Evaluation Benchmarks](https://mlsanity.com/paper/2406.10229/) (2024)
- [Credal Bayesian Deep Learning](https://mlsanity.com/paper/2302.09656/) (2023)
- [DeepOBS: A Deep Learning Optimizer Benchmark Suite](https://mlsanity.com/paper/1903.05499/) (2019)
- [A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2111.06784/) (2021)

## Built on by (most cited first, 9 of 9)

- [Do Repetitions Matter? Strengthening Reliability in LLM Evaluations](https://mlsanity.com/paper/2509.24086/) (2025)
- [The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Models](https://mlsanity.com/paper/2509.23088/) (2025)
- [The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices](https://mlsanity.com/paper/2603.18482/) (2026)
- [Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion](https://mlsanity.com/paper/2605.22579/) (2026)
- [Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding](https://mlsanity.com/paper/2606.22511/) (2026)
- [Empirical Decision Theory](https://mlsanity.com/paper/2512.05677/) (2025)
- [Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2605.12208/) (2026)
- [The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic](https://mlsanity.com/paper/2605.28700/) (2026)
- [Reference-Based Analysis of Coherence and Diversity in Open-Ended Text Generation](https://mlsanity.com/paper/2609.28080/) (2026)

## Abstract (arXiv)

Assessing the quality of LLM-generated text remains a fundamental challenge in natural language processing. Current evaluation approaches often rely on isolated metrics or simplistic aggregations that fail to capture the nuanced trade-offs between coherence, diversity, fluency, and other relevant indicators of text quality. In this work, we adapt a recently proposed framework for statistical inference based on Generalized Stochastic Dominance (GSD) that addresses three critical limitations in existing benchmarking methodologies: the inadequacy of single-metric evaluation, the incompatibility between cardinal automatic metrics and ordinal human judgments, and the lack of inferential statistical guarantees. The GSD-front approach enables simultaneous evaluation across multiple quality dimensions while respecting their different measurement scales, building upon partial orders of decoding strategies, thus avoiding arbitrary weighting of the involved metrics. By applying this framework to evaluate common decoding strategies against human-generated text, we demonstrate its ability to identify statistically significant performance differences while accounting for potential deviations from the i.i.d. assumption of the sampling design.