# Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?

arXiv: https://arxiv.org/abs/2502.07963 · HTML: https://mlsanity.com/paper/2502.07963/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 81 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [MathPrompter: Mathematical Reasoning using Large Language Models](https://mlsanity.com/paper/2303.05398/) (2023)
- [ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports](https://mlsanity.com/paper/2212.14882/) (2022)
- [BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine](https://mlsanity.com/paper/2308.09442/) (2023)
- [AlpaCare:Instruction-tuned Large Language Models for Medical Application](https://mlsanity.com/paper/2310.14558/) (2023)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [Paragraph-level Simplification of Medical Texts](https://mlsanity.com/paper/2104.05767/) (2021)
- [Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)](https://mlsanity.com/paper/2305.06299/) (2023)
- [Evaluating Factuality in Text Simplification](https://mlsanity.com/paper/2204.07562/) (2022)
- [Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?](https://mlsanity.com/paper/2411.04118/) (2024)
- [Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing](https://mlsanity.com/paper/2203.00130/) (2022)
- [FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence](https://mlsanity.com/paper/2402.11456/) (2024)
- [Multilingual Simplification of Medical Texts](https://mlsanity.com/paper/2305.12532/) (2023)
- [Leveraging Large Language Models for Zero-shot Lay Summarisation in Biomedicine and Beyond](https://mlsanity.com/paper/2501.05224/) (2025)

## Built on by (most cited first, 3 of 3)

- [A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments](https://mlsanity.com/paper/2509.25609/) (2025)
- [Beyond Text: Characterizing Domain Expert Needs in Document Research](https://mlsanity.com/paper/2504.12495/) (2025)
- [Position: Behavioral Systems Require Behavioral Tests](https://mlsanity.com/paper/2608.18081/) (2026)

## Abstract (arXiv)

Medical research faces well-documented challenges in translating novel treatments into clinical practice. Publishing incentives encourage researchers to present "positive" findings, even when empirical results are equivocal. Consequently, it is well-documented that authors often spin study results, especially in article abstracts. Such spin can influence clinician interpretation of evidence and may affect patient care decisions. In this study, we ask whether the interpretation of trial results offered by Large Language Models (LLMs) is similarly affected by spin. This is important since LLMs are increasingly being used to trawl through and synthesize published medical evidence. We evaluated 22 LLMs and found that they are across the board more susceptible to spin than humans. They might also propagate spin into their outputs: We find evidence, e.g., that LLMs implicitly incorporate spin into plain language summaries that they generate. We also find, however, that LLMs are generally capable of recognizing spin, and can be prompted in a way to mitigate spin's impact on LLM outputs.