# Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks

arXiv: https://arxiv.org/abs/2601.21339 · HTML: https://mlsanity.com/paper/2601.21339/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 19 of 19)

- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance](https://mlsanity.com/paper/2401.03729/) (2024)
- [Is Temperature the Creativity Parameter of Large Language Models?](https://mlsanity.com/paper/2405.00492/) (2024)
- [When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models](https://mlsanity.com/paper/2311.10054/) (2023)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [On the Worst Prompt Performance of Large Language Models](https://mlsanity.com/paper/2406.10248/) (2024)
- [Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers](https://mlsanity.com/paper/2310.02905/) (2023)
- [Creativity Has Left the Chat: The Price of Debiasing Language Models](https://mlsanity.com/paper/2406.05587/) (2024)
- [Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks](https://mlsanity.com/paper/2412.15238/) (2024)
- [Devil's Advocate: Anticipatory Reflection for LLM Agents](https://mlsanity.com/paper/2405.16334/) (2024)
- [LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction](https://mlsanity.com/paper/2311.06555/) (2023)
- [Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variability](https://mlsanity.com/paper/2504.12320/) (2025)
- [Code Roulette: How Prompt Variability Affects LLM Code Generation](https://mlsanity.com/paper/2506.10204/) (2025)

## Built on by (most cited first, 3 of 3)

- [Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking](https://mlsanity.com/paper/2604.11581/) (2026)
- [CAFE: A Compound-AI Factorial Evaluation Framework](https://mlsanity.com/paper/2607.10380/) (2026)
- [PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models](https://mlsanity.com/paper/2609.34195/) (2026)

## Abstract (arXiv)

How much of LLM output variance is explained by prompts versus model choice versus stochasticity through sampling? We answer this by evaluating 12 LLMs on 10 creativity prompts with 100 samples each (N = 12,000). For output quality (originality), prompts explain 36.43% of variance, comparable to model choice (40.94%). But for output quantity (fluency), model choice (51.25%) and within-LLM variance (33.70%) dominate, with prompts explaining only 4.22%. Prompts are powerful levers for steering output quality, but given the substantial within-LLM variance (10-34%), single-sample evaluations risk conflating sampling noise with genuine prompt or model effects.