# Can Large Language Models Detect Misinformation in Scientific News Reporting? arXiv: https://arxiv.org/abs/2402.14268 ยท HTML: https://mlsanity.com/paper/2402.14268/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 58 references that are in the map. - First posted on arXiv 22 Feb 2024. ## Builds on (most cited first, 25 of 25) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019) - [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019) - [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018) - [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023) - [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023) - [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023) - [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021) - ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017) - [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020) - [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020) - [Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://mlsanity.com/paper/2302.08081/) (2023) - [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021) - [SciFact-Open: Towards open-domain scientific claim verification](https://mlsanity.com/paper/2210.13777/) (2022) - [Generating Scientific Claims for Zero-Shot Scientific Fact Checking](https://mlsanity.com/paper/2203.12990/) (2022) - [Extractive Summarization via ChatGPT for Faithful Summary Generation](https://mlsanity.com/paper/2304.04193/) (2023) - [Scientific Fact-Checking: A Survey of Resources and Approaches](https://mlsanity.com/paper/2305.16859/) (2023) - [Scientific Claim Verification with VERT5ERINI](https://mlsanity.com/paper/2010.11930/) (2020) - [Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications](https://mlsanity.com/paper/2109.14776/) (2021) - [Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence](https://mlsanity.com/paper/2305.18265/) (2023) - [Overview and Insights from the SciVer Shared Task on Scientific Claim Verification](https://mlsanity.com/paper/2107.08188/) (2021) - [A Corpus of Sentence-level Revisions in Academic Writing: A Step towards Understanding Statement Strength in Communication](https://mlsanity.com/paper/1405.1439/) (2014) - [Modeling Information Change in Science Communication with Semantically Matched Paraphrases](https://mlsanity.com/paper/2210.13001/) (2022) - [MMCoVaR: Multimodal COVID-19 Vaccine Focused Data Repository for Fake News Detection and a Baseline Architecture for Classification](https://mlsanity.com/paper/2109.06416/) (2021) - [Semi-Supervised Exaggeration Detection of Health Science Press Releases](https://mlsanity.com/paper/2108.13493/) (2021) ## Built on by (most cited first, 9 of 9) - [Claim Verification in the Age of Large Language Models: A Survey](https://mlsanity.com/paper/2408.14317/) (2024) - [Unmasking Digital Falsehoods: A Comparative Analysis of LLM-Based Misinformation Detection Strategies](https://mlsanity.com/paper/2503.00724/) (2025) - [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025) - [Pub-Guard-LLM: Detecting Retracted Biomedical Articles with Reliable Explanations](https://mlsanity.com/paper/2502.15429/) (2025) - [Bonsai: Interpretable Tree-Adaptive Grounded Reasoning](https://mlsanity.com/paper/2504.03640/) (2025) - [Toward Reliable, Safe, and Secure LLMs for Scientific Applications](https://mlsanity.com/paper/2603.18235/) (2026) - [LLM Analysis of 150+ years of German Parliamentary Debates on Migration Reveals Shift from Post-War Solidarity to Anti-Solidarity in the Last Decade](https://mlsanity.com/paper/2509.07274/) (2025) - [Are LLMs More Skeptical of Entertainment News?](https://mlsanity.com/paper/2605.01727/) (2026) - [From Generation to Detection: Exploration of Discourse Driven Scenario based LLM Generated Fake News](https://mlsanity.com/paper/2609.20838/) (2026) ## Abstract (arXiv) Scientific facts are often spun in the popular press with the intent to influence public opinion and action, as was evidenced during the COVID-19 pandemic. Automatic detection of misinformation in the scientific domain is challenging because of the distinct styles of writing in these two media types and is still in its nascence. Most research on the validity of scientific reporting treats this problem as a claim verification challenge. In doing so, significant expert human effort is required to generate appropriate claims. Our solution bypasses this step and addresses a more real-world scenario where such explicit, labeled claims may not be available. The central research question of this paper is whether it is possible to use large language models (LLMs) to detect misinformation in scientific reporting. To this end, we first present a new labeled dataset SciNews, containing 2.4k scientific news stories drawn from trusted and untrustworthy sources, paired with related abstracts from the CORD-19 database. Our dataset includes both human-written and LLM-generated news articles, making it more comprehensive in terms of capturing the growing trend of using LLMs to generate popular press articles. Then, we identify dimensions of scientific validity in science news articles and explore how this can be integrated into the automated detection of scientific misinformation. We propose several baseline architectures using LLMs to automatically detect false representations of scientific findings in the popular press. For each of these architectures, we use several prompt engineering strategies including zero-shot, few-shot, and chain-of-thought prompting. We also test these architectures and prompting strategies on GPT-3.5, GPT-4, and Llama2-7B, Llama2-13B.