# Fact-checking information from large language models can decrease headline discernment arXiv: https://arxiv.org/abs/2308.10800 ยท HTML: https://mlsanity.com/paper/2308.10800/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Builds on 15 of its 81 references that are in the map. - First posted on arXiv 21 Aug 2023. ## Builds on (most cited first, 15 of 15) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023) - [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020) - [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021) - [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017) - [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020) - [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021) - [The science of fake news](https://mlsanity.com/paper/2307.07903/) (2023) - [Explainable Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2011.03870/) (2020) - [AI model GPT-3 (dis)informs us better than humans](https://mlsanity.com/paper/2301.11924/) (2023) - [Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2109.11427/) (2021) - [Reinforcement Learning-based Counter-Misinformation Response Generation: A Case Study of COVID-19 Vaccine Misinformation](https://mlsanity.com/paper/2303.06433/) (2023) - [Anatomy of an AI-powered malicious social botnet](https://mlsanity.com/paper/2307.16336/) (2023) - [Online misinformation is linked to early COVID-19 vaccination hesitancy and refusal](https://mlsanity.com/paper/2104.10635/) (2021) - [The Effects of Interactive AI Design on User Behavior: An Eye-tracking Study of Fact-checking COVID-19 Claims](https://mlsanity.com/paper/2202.08901/) (2022) ## Built on by (most cited first, 22 of 22) - [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023) - [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025) - [Epistemological Fault Lines Between Human and Artificial Intelligence](https://mlsanity.com/paper/2512.19466/) (2025) - [Customized large language models can outperform Community Notes in correcting misinformation](https://mlsanity.com/paper/2403.11169/) (2024) - [Refusal Steering: Fine-grained Control over LLM Refusal Behaviour for Sensitive Topics](https://mlsanity.com/paper/2512.16602/) (2025) - [AI Coding Agents Can Reproduce Social Science Findings](https://mlsanity.com/paper/2606.11447/) (2026) - [Community Moderation and the New Epistemology of Fact Checking on Social Media](https://mlsanity.com/paper/2505.20067/) (2025) - [From Knowledge Generation to Knowledge Verification: Examining the BioMedical Generative Capabilities of ChatGPT](https://mlsanity.com/paper/2502.14714/) (2025) - [How LLMs Fail to Support Fact-Checking](https://mlsanity.com/paper/2503.01902/) (2025) - [Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Search](https://mlsanity.com/paper/2511.18749/) (2025) - [Truth Sleuth and Trend Bender: AI Agents to fact-check YouTube videos and influence opinions](https://mlsanity.com/paper/2507.10577/) (2025) - [Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning](https://mlsanity.com/paper/2510.18254/) (2025) - [Understanding Inequality of LLM Fact-Checking over Geographic Regions with Agent and Retrieval models](https://mlsanity.com/paper/2503.22877/) (2025) - [Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods](https://mlsanity.com/paper/2505.19402/) (2025) - [From prosthetic memory to prosthetic denial: Auditing whether large language models are prone to mass atrocity denialism](https://mlsanity.com/paper/2505.21753/) (2025) - [Do Androids Dream of Unseen Puppeteers? Probing for a Conspiracy Tendencies in Large Language Models](https://mlsanity.com/paper/2511.03699/) (2025) - [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026) - [Train Yourself as an LLM: Exploring Effects of AI Literacy on Persuasion via Role-playing LLM Training](https://mlsanity.com/paper/2604.02637/) (2026) - [Generative AI and Digital Ecosystem Resilience: A Proactive Lifecycle-Based Survey](https://mlsanity.com/paper/2606.00136/) (2026) - [Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability](https://mlsanity.com/paper/2607.10402/) (2026) - [Unequal Verdicts: Investigating Gender Bias in LLM-Based Fake News Detection](https://mlsanity.com/paper/2608.03627/) (2026) - [Epistemic Transfer in AI-Assisted Verification: A Framework and Evaluation Protocol](https://mlsanity.com/paper/2608.08882/) (2026) ## Abstract (arXiv) Fact checking can be an effective strategy against misinformation, but its implementation at scale is impeded by the overwhelming volume of information online. Recent artificial intelligence (AI) language models have shown impressive ability in fact-checking tasks, but how humans interact with fact-checking information provided by these models is unclear. Here, we investigate the impact of fact-checking information generated by a popular large language model (LLM) on belief in, and sharing intent of, political news headlines in a preregistered randomized control experiment. Although the LLM accurately identifies most false headlines (90%), we find that this information does not significantly improve participants' ability to discern headline accuracy or share accurate news. In contrast, viewing human-generated fact checks enhances discernment in both cases. Subsequent analysis reveals that the AI fact-checker is harmful in specific cases: it decreases beliefs in true headlines that it mislabels as false and increases beliefs in false headlines that it is unsure about. On the positive side, AI fact-checking information increases the sharing intent for correctly labeled true headlines. When participants are given the option to view LLM fact checks and choose to do so, they are significantly more likely to share both true and false news but only more likely to believe false headlines. Our findings highlight an important source of potential harm stemming from AI applications and underscore the critical need for policies to prevent or mitigate such unintended consequences.