# Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making

arXiv: https://arxiv.org/abs/2508.15926 · HTML: https://mlsanity.com/paper/2508.15926/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 58 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Builds on (most cited first, 23 of 23)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements](https://mlsanity.com/paper/2402.10963/) (2024)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena](https://mlsanity.com/paper/2310.05746/) (2023)
- [Do LLM Agents Exhibit Social Behavior?](https://mlsanity.com/paper/2312.15198/) (2023)
- [LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory](https://mlsanity.com/paper/2408.02784/) (2024)
- [From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models](https://mlsanity.com/paper/2308.12014/) (2023)
- [Can Large Language Models Identify Authorship?](https://mlsanity.com/paper/2403.08213/) (2024)
- [Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context](https://mlsanity.com/paper/2406.05972/) (2024)
- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs](https://mlsanity.com/paper/2406.11278/) (2024)
- [RTBAgent: A LLM-based Agent System for Real-Time Bidding](https://mlsanity.com/paper/2502.00792/) (2025)
- [LLM Questionnaire Completion for Automatic Psychiatric Assessment](https://mlsanity.com/paper/2406.06636/) (2024)
- [Can AI with High Reasoning Ability Replicate Human-like Decision Making in Economic Experiments?](https://mlsanity.com/paper/2406.11426/) (2024)
- [Can Machines Think Like Humans? A Behavioral Evaluation of LLM Agents in Dictator Games](https://mlsanity.com/paper/2410.21359/) (2024)
- [Can LLMs Learn Macroeconomic Narratives from Social Media?](https://mlsanity.com/paper/2406.12109/) (2024)
- [XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs](https://mlsanity.com/paper/2311.08614/) (2023)
- [Top-nσ: Not All Logits Are You Need](https://mlsanity.com/paper/2411.07641/) (2024)

## Built on by (most cited first, 4 of 4)

- [ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders](https://mlsanity.com/paper/2602.16938/) (2026)
- [Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices](https://mlsanity.com/paper/2602.09802/) (2026)
- [HAG: Hierarchical Demographic Tree-based Agent Generation for Topic-Adaptive Simulation](https://mlsanity.com/paper/2601.05656/) (2026)
- [LLMs Should Not Yet Be Credited with Decision Explanation](https://mlsanity.com/paper/2605.01164/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used in social science simulations. While their performance on reasoning and optimization tasks has been extensively evaluated, less attention has been paid to their ability to simulate human decision-making's variability and adaptability. We propose a process-oriented evaluation framework with progressive interventions (Intrinsicality, Instruction, and Imitation) to examine how LLM agents adapt under different levels of external guidance and human-derived noise. We validate the framework on two classic economics tasks, irrationality in the second-price auction and decision bias in the newsvendor problem, showing behavioral gaps between LLMs and humans. We find that LLMs, by default, converge on stable and conservative strategies that diverge from observed human behaviors. Risk-framed instructions impact LLM behavior predictably but do not replicate human-like diversity. Incorporating human data through in-context learning narrows the gap but fails to reach human subjects' strategic variability. These results highlight a persistent alignment gap in behavioral fidelity and suggest that future LLM evaluations should consider more process-level realism. We present a process-oriented approach for assessing LLMs in dynamic decision-making tasks, offering guidance for their application in synthetic data for social science research.