mlsanityData to 14 Oct 2026

Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework

arXiv 2410.18653 ↗First posted 24 Oct 2024cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Open-ended text generation has become a prominent task in natural language processing due to the rise of powerful (large) language models. However, evaluating the quality of these models and the employed decoding strategies remains challenging due to trade-offs among widely used metrics such as coherence, diversity, and perplexity. This paper addresses the specific problem of multicriteria evaluation for open-ended text generation, proposing novel methods for both relative and absolute rankings of decoding methods. Specifically, we employ benchmarking approaches based on partial orderings and present a new summary metric to balance existing automatic indicators, providing a more holistic evaluation of text generation quality. Our experiments demonstrate that the proposed approaches offer a robust way to compare decoding strategies and serve as valuable tools to guide model selection for open-ended text generation tasks. We suggest future directions for improving evaluation methodologies in text generation and make our code, datasets, and models publicly available.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

34 of 69 references.

References · 34

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  5. 2018Hierarchical Neural Story Generationcited by 1,193
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  9. 2014OpenML: networked science in machine learningcited by 526
  10. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  11. 2020Evaluation of Text Generation: A Surveycited by 277
  12. 2017Beam Search Strategies for Neural Machine Translationcited by 232
22 more
  1. 2022A Contrastive Framework for Neural Text Generationcited by 225
  2. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
  3. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  4. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  5. 2016Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysiscited by 93
  6. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  7. 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
  8. 2019Jointly Measuring Diversity and Quality in Text Generation Modelscited by 61
  9. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 59
  10. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  11. 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
  12. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 28
  13. 2024Evaluating language models as risk scorescited by 27
  14. 2024Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generationcited by 27
  15. 2024Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generationcited by 26
  16. 2022Statistical Comparisons of Classifiers by Generalized Stochastic Dominancecited by 20
  17. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  18. 2023Robust Statistical Comparison of Random Variables with Locally Varying Scale of Measurementcited by 13
  19. 2024Rethinking Generative Large Language Model Evaluation for Semantic Comprehensioncited by 13
  20. 2024Statistical Multicriteria Benchmarking via the GSD-Frontcited by 11
  21. 2024Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluationcited by 5
  22. 2023Comparing Machine Learning Algorithms by Union-Free Generic Depthcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025GUARD: Glocal Uncertainty-Aware Robust Decoding for Effective and Efficient Open-Ended Text Generationcited by 10
  2. 2025Statistical Multicriteria Evaluation of LLM-Generated Textcited by 9
  3. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  4. 2025The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Modelscited by 4
  5. 2026The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choicescited by 3
  6. 2026Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansioncited by 1
  7. 2025Empirical Decision Theorycited by 0
  8. 2026Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantificationcited by 0
  9. 2026Reference-Based Analysis of Coherence and Diversity in Open-Ended Text Generationcited by 0