mlsanityData to 14 Oct 2026

The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Models

arXiv 2509.23088 ↗First posted 27 Sep 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Understanding uncertainty in large language models remains a fundamental challenge, particularly in creative tasks where multiple valid outputs exist. We present a geometric framework using credal sets - convex hulls of probability distributions - to quantify and decompose uncertainty in neural text generation, calibrated against human creative variation. Analyzing 500 creative writing prompts from the WritingPrompts dataset with 10 unique human continuations each, we evaluate four language models across five decoding strategies, generating 100,000 stories. Our credal set analysis reveals substantial gaps in capturing human creative variation, with the best model-human calibration reaching only 0.434 (Gemma-2B with temperature 0.7). We decompose total uncertainty into epistemic and aleatoric components, finding that the choice of decoding strategy contributes 39.4% to 72.0% of total epistemic uncertainty. Model scale shows weak correlation with calibration quality and no significant difference exists between base and instruction-tuned models in calibration quality. Our geometric framework provides actionable insights for improving generation systems for human-AI creative alignment. We release our complete experimental framework.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

21 of 33 references.

References · 21

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  7. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  8. 2018Hierarchical Neural Story Generationcited by 1,193
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  11. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 125
  12. 2024To Believe or Not to Believe Your LLMcited by 120
9 more
  1. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 114
  2. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 76
  3. 2024Calibrating Large Language Models Using Their Generations Onlycited by 65
  4. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  5. 2024Uncertainty Quantification for In-Context Learning of Large Language Modelscited by 45
  6. 2024Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generationcited by 27
  7. 2025GUARD: Glocal Uncertainty-Aware Robust Decoding for Effective and Efficient Open-Ended Text Generationcited by 10
  8. 2024Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Frameworkcited by 9
  9. 2025Statistical Multicriteria Evaluation of LLM-Generated Textcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writerscited by 4
  2. 2026The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choicescited by 3
  3. 2026Credal Large Language Models for Semantic Commitment under Uncertaintycited by 2
  4. 2026Reference-Based Analysis of Coherence and Diversity in Open-Ended Text Generationcited by 0