# Generalization Bias in Large Language Model Summarization of Scientific Research

arXiv: https://arxiv.org/abs/2504.00025 · HTML: https://mlsanity.com/paper/2504.00025/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 67 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 14 of 14)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?](https://mlsanity.com/paper/2405.16908/) (2024)
- [FABLES: Evaluating faithfulness and content selection in book-length summarization](https://mlsanity.com/paper/2404.01261/) (2024)
- [Calibrating the Confidence of Large Language Models by Eliciting Fidelity](https://mlsanity.com/paper/2404.02655/) (2024)
- [Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential](https://mlsanity.com/paper/2303.09038/) (2023)
- [Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledge](https://mlsanity.com/paper/2311.09731/) (2023)
- [Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters](https://mlsanity.com/paper/2312.17476/) (2023)
- [ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor](https://mlsanity.com/paper/2406.14765/) (2024)
- [Anthropomorphization of AI: Opportunities and Risks](https://mlsanity.com/paper/2305.14784/) (2023)
- [A Sign That Spells: DALL-E 2, Invisual Images and The Racial Politics of Feature Space](https://mlsanity.com/paper/2211.06323/) (2022)

## Built on by (most cited first, 30 of 30)

- [Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM](https://mlsanity.com/paper/2508.05775/) (2025)
- [First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation](https://mlsanity.com/paper/2511.04715/) (2025)
- [A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation](https://mlsanity.com/paper/2608.24419/) (2026)
- [Human-AI Interactions: Cognitive, Behavioral, and Emotional Impacts](https://mlsanity.com/paper/2510.17753/) (2025)
- [LINK-KG: LLM-Driven Coreference-Resolved Knowledge Graphs for Human Smuggling Networks](https://mlsanity.com/paper/2510.26486/) (2025)
- [Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs](https://mlsanity.com/paper/2510.26512/) (2025)
- [When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis](https://mlsanity.com/paper/2606.29251/) (2026)
- [CORE-KG: An LLM-Driven Knowledge Graph Construction Framework for Human Smuggling Networks](https://mlsanity.com/paper/2506.21607/) (2025)
- [Why can't Epidemiology be automated (yet)?](https://mlsanity.com/paper/2507.15617/) (2025)
- [Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning](https://mlsanity.com/paper/2510.18254/) (2025)
- [Faithful Summarisation under Disagreement via Belief-Level Aggregation](https://mlsanity.com/paper/2601.04889/) (2026)
- [Generics in science communication: Misaligned interpretations across laypeople, scientists, and large language models](https://mlsanity.com/paper/2602.06190/) (2026)
- [Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses](https://mlsanity.com/paper/2606.01845/) (2026)
- [Fair Representation in Parliamentary Summaries: Measuring and Mitigating Inclusion Bias](https://mlsanity.com/paper/2507.14221/) (2025)
- [Towards Personalized Explanations for Health Simulations: A Mixed-Methods Framework for Stakeholder-Centric Summarization](https://mlsanity.com/paper/2509.04646/) (2025)
- [The Prompt Engineering Report Distilled: Quick Start Guide for Life Sciences](https://mlsanity.com/paper/2509.11295/) (2025)
- [A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving](https://mlsanity.com/paper/2509.23384/) (2025)
- ['Too much alignment; not enough culture': Re-balancing cultural alignment practices in LLMs](https://mlsanity.com/paper/2509.26167/) (2025)
- [DiscoVerse: Multi-Agent Pharmaceutical Co-Scientist for Traceable Drug Discovery and Reverse Translation](https://mlsanity.com/paper/2511.18259/) (2025)
- [The Effect of Document Summarization on LLM-Based Relevance Judgments](https://mlsanity.com/paper/2512.05334/) (2025)
- [LLM or Human? Perceptions of Trust and Information Quality in Research Summaries](https://mlsanity.com/paper/2601.15556/) (2026)
- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)
- [Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech](https://mlsanity.com/paper/2605.17652/) (2026)
- [What Do People Actually Want From AI? Mapping Preference Plurality](https://mlsanity.com/paper/2606.06674/) (2026)
- [AI-Assisted Computational Reproducibility on the FABRIC Testbed](https://mlsanity.com/paper/2606.25879/) (2026)
- [What is Left for Us? Second Scholarship Against the Degradation of Research by AI](https://mlsanity.com/paper/2607.04049/) (2026)
- [When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation](https://mlsanity.com/paper/2608.19083/) (2026)
- [Performance of a domain-specific large language model in answering patient questions in psychiatry](https://mlsanity.com/paper/2608.22797/) (2026)
- [ARISMA: Guidelines for AI- and LLM-Assisted Systematic Reviews, Scoping Reviews, and Mapping Studies](https://mlsanity.com/paper/2608.25050/) (2026)
- [Editorial routing shapes how computational results are qualified in AI-assisted scientific writing](https://mlsanity.com/paper/2609.14288/) (2026)

## Abstract (arXiv)

Artificial intelligence chatbots driven by large language models (LLMs) have the potential to increase public science literacy and support scientific research, as they can quickly summarize complex scientific information in accessible terms. However, when summarizing scientific texts, LLMs may omit details that limit the scope of research conclusions, leading to generalizations of results broader than warranted by the original study. We tested 10 prominent LLMs, including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3 70B, and Claude 3.7 Sonnet, comparing 4900 LLM-generated summaries to their original scientific texts. Even when explicitly prompted for accuracy, most LLMs produced broader generalizations of scientific results than those in the original texts, with DeepSeek, ChatGPT-4o, and LLaMA 3.3 70B overgeneralizing in 26 to 73% of cases. In a direct comparison of LLM-generated and human-authored science summaries, LLM summaries were nearly five times more likely to contain broad generalizations (OR = 4.85, 95% CI [3.06, 7.70]). Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings. We highlight potential mitigation strategies, including lowering LLM temperature settings and benchmarking LLMs for generalization accuracy.