# Reference-Based Analysis of Coherence and Diversity in Open-Ended Text Generation

arXiv: https://arxiv.org/abs/2609.28080 · HTML: https://mlsanity.com/paper/2609.28080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics](https://mlsanity.com/paper/2105.08920/) (2021)
- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation](https://mlsanity.com/paper/2407.18698/) (2024)
- [Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation](https://mlsanity.com/paper/2410.06097/) (2024)
- [GUARD: Glocal Uncertainty-Aware Robust Decoding for Effective and Efficient Open-Ended Text Generation](https://mlsanity.com/paper/2508.20757/) (2025)
- [Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework](https://mlsanity.com/paper/2410.18653/) (2024)
- [Statistical Multicriteria Evaluation of LLM-Generated Text](https://mlsanity.com/paper/2506.18082/) (2025)
- [Min-k Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics](https://mlsanity.com/paper/2604.11012/) (2026)
- [The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Models](https://mlsanity.com/paper/2509.23088/) (2025)
- [The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices](https://mlsanity.com/paper/2603.18482/) (2026)
- [Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding](https://mlsanity.com/paper/2606.22511/) (2026)

## Abstract (arXiv)

Evaluating open-ended text generation involves understanding how different properties of a continuation relate to its perceived quality. We present a reference-based framework for examining coherence and diversity through three perspectives: aligning their evolution with human trajectories, comparing their summaries with a human continuation of the same prompt, and estimating their likelihood under a human reference distribution. Experiments with human quality ratings suggest that diversity-based alignment and mean-based comparisons capture quality-related variation, although the comparisons do not establish a predictive advantage for temporal alignment over simpler baselines. Reference likelihood also shows positive associations with ratings, with results varying across reference configurations and scoring horizons. Together, these analyses provide a structured way to examine how measured coherence and diversity relate to human judgments, while distinguishing similarity to human references from quality itself. Code and analysis resources are available at https://github.com/EstebanGarces/likely_human.