mlsanityData to 14 Oct 2026

Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks

arXiv 2601.21339 ↗First posted 29 Jan 2026cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

How much of LLM output variance is explained by prompts versus model choice versus stochasticity through sampling? We answer this by evaluating 12 LLMs on 10 creativity prompts with 100 samples each (N = 12,000). For output quality (originality), prompts explain 36.43% of variance, comparable to model choice (40.94%). But for output quantity (fluency), model choice (51.25%) and within-LLM variance (33.70%) dominate, with prompts explaining only 4.22%. Prompts are powerful levers for steering output quality, but given the substantial within-LLM variance (10-34%), single-sample evaluations risk conflating sampling noise with genuine prompt or model effects.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

19 of 28 references.

References · 19

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  2. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  3. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  4. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  5. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  6. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  7. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  8. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 100
  9. 2024Is Temperature the Creativity Parameter of Large Language Models?cited by 93
  10. 2023When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Modelscited by 53
  11. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 51
  12. 2024On the Worst Prompt Performance of Large Language Modelscited by 38
7 more
  1. 2023Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformerscited by 30
  2. 2024Creativity Has Left the Chat: The Price of Debiasing Language Modelscited by 29
  3. 2024Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning taskscited by 22
  4. 2024Devil's Advocate: Anticipatory Reflection for LLM Agentscited by 17
  5. 2023LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extractioncited by 8
  6. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  7. 2025Code Roulette: How Prompt Variability Affects LLM Code Generationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarkingcited by 10
  2. 2026CAFE: A Compound-AI Factorial Evaluation Frameworkcited by 1
  3. 2026PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Modelscited by 0