# Comparing the Framing Effect in Humans and LLMs on Naturally Occurring Texts

arXiv: https://arxiv.org/abs/2502.17091 · HTML: https://mlsanity.com/paper/2502.17091/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing](https://mlsanity.com/paper/2309.05898/) (2023)
- [A Comprehensive Evaluation of Cognitive Biases in LLMs](https://mlsanity.com/paper/2410.15413/) (2024)
- [CBEval: A framework for evaluating and interpreting cognitive biases in LLMs](https://mlsanity.com/paper/2412.03605/) (2024)
- [Interactive Extractive Search over Biomedical Corpora](https://mlsanity.com/paper/2006.04148/) (2020)

## Built on by (most cited first, 6 of 6)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs](https://mlsanity.com/paper/2507.07186/) (2025)
- [Getting out of the Big-Muddy: Escalation of Commitment in LLMs](https://mlsanity.com/paper/2508.01545/) (2025)
- [Stability Regimes for Framing-Sensitive Fine-Tuning in Language Models](https://mlsanity.com/paper/2602.15273/) (2026)
- [Confirming Our Biases? Evaluating the Capabilities, Risks, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2608.06977/) (2026)
- [Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Making](https://mlsanity.com/paper/2609.22225/) (2026)

## Abstract (arXiv)

Humans are influenced by how information is presented, a phenomenon known as the framing effect. Prior work suggests that LLMs may also be susceptible to framing, but it has relied on synthetic data and did not compare to human behavior. To address this gap, we introduce WildFrame - a dataset for evaluating LLM responses to positive and negative framing in naturally-occurring sentences, alongside human responses on the same data. WildFrame consists of 1,000 real-world texts selected to convey a clear sentiment; we then reframe each text in either a positive or negative light and collect human sentiment annotations. Evaluating eleven LLMs on WildFrame, we find that all models respond to reframing in a human-like manner (r≥0.52), and that both humans and models are influenced more by positive than negative reframing. Notably, GPT models are the least correlated with human behavior among all tested models. These findings raise a discussion around the goals of state-of-the-art LLM development and whether models should align closely with human behavior, to preserve cognitive phenomena such as the framing effect, or instead mitigate such biases in favor of fairness and consistency.