mlsanityData to 14 Oct 2026

Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters

arXiv 2312.17476 ↗First posted 29 Dec 2023cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

The advancement of Large Language Models (LLMs) has led to their widespread use across a broad spectrum of tasks including decision making. Prior studies have compared the decision making abilities of LLMs with those of humans from a psychological perspective. However, these studies have not always properly accounted for the sensitivity of LLMs' behavior to hyperparameters and variations in the prompt. In this study, we examine LLMs' performance on the Horizon decision making task studied by Binz and Schulz (2023) analyzing how LLMs respond to variations in prompts and hyperparameters. By experimenting on three OpenAI language models possessing different capabilities, we observe that the decision making abilities fluctuate based on the input prompts and temperature settings. Contrary to previous findings language models display a human-like exploration exploitation tradeoff after simple adjustments to the prompt.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 12 of its 22 references that are in the map. First posted on arXiv 29 Dec 2023.

Builds on

12 of 22 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2022Using cognitive psychology to understand GPT-3cited by 398
  6. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  7. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  8. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  9. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
  10. 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
  11. 2023Do large language models resemble humans in language use?cited by 34
  12. 2023The Machine Psychology of Cooperation: Can GPT models operationalise prompts for altruism, cooperation, competitiveness and selfishness in economic games?cited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 13; 2026: 5.

32 papers.

Sort:

By area:

  1. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  2. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  3. 2025Benchmarking Prompt Sensitivity in Large Language Modelscited by 38
  4. 2025Generalization Bias in Large Language Model Summarization of Scientific Researchcited by 30
  5. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  6. 2024The Better Angels of Machine Personality: How Personality Relates to LLM Safetycited by 21
  7. 2025Literature-Grounded Novelty Assessment of Scientific Ideascited by 19
  8. 2025An overview of model uncertainty and variability in LLM-based sentiment analysis. Challenges, mitigation strategies and the role of explainabilitycited by 15
  9. 2024From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMscited by 13
  10. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 13
  11. 2024Explaining Large Language Models Decisions Using Shapley Valuescited by 12
  12. 2026One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalizationcited by 9
20 more
  1. 2025Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Modelscited by 7
  2. 2024Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metriccited by 6
  3. 2025Acquiescence Bias in Large Language Modelscited by 5
  4. 2024Evaluating and Enhancing Trustworthiness of LLMs in Perception Taskscited by 4
  5. 2024Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfittingcited by 4
  6. 2024GLaPE: Gold Label-agnostic Prompt Evaluation and Optimization for Large Language Modelcited by 3
  7. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Modelscited by 3
  8. 2025LLM Agents Are the Antidote to Walled Gardenscited by 3
  9. 2024Enhancing Multi-Criteria Decision Analysis with AI: Integrating Analytic Hierarchy Process and GPT-4 for Automated Decision Supportcited by 2
  10. 2024Irrelevant Alternatives Bias Large Language Model Hiring Decisionscited by 2
  11. 2025Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysiscited by 2
  12. 2026HumanStudy-Bench: Towards AI Agent Design for Participant Simulationcited by 2
  13. 2026This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QAcited by 2
  14. 2024GenCeption: Evaluate Vision LLMs with Unlabeled Unimodal Datacited by 1
  15. 2024PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematicscited by 1
  16. 2025Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methodscited by 1
  17. 2026Assessing mentalization in humans and large language modelscited by 1
  18. 2025Understanding Human Limits in Pattern Recognition: A Computational Model of Sequential Reasoning in Rock, Paper, Scissorscited by 0
  19. 2025Rethinking Reasoning in LLMs: Neuro-Symbolic Local RetoMaton Beyond ICL and CoTcited by 0
  20. 2026Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Modelscited by 0