# Fact-checking AI-generated news reports: Can LLMs catch their own lies?

arXiv: https://arxiv.org/abs/2503.18293 · HTML: https://mlsanity.com/paper/2503.18293/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 14 of 14)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification](https://mlsanity.com/paper/2403.04696/) (2024)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [Decontextualization: Making Sentences Stand-Alone](https://mlsanity.com/paper/2102.05169/) (2021)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [The Automated Verification of Textual Claims (AVeriTeC) Shared Task](https://mlsanity.com/paper/2410.23850/) (2024)
- [Question-Based Retrieval using Atomic Units for Enterprise RAG](https://mlsanity.com/paper/2405.12363/) (2024)
- [AI "News" Content Farms Are Easy to Make and Hard to Detect: A Case Study in Italian](https://mlsanity.com/paper/2406.12128/) (2024)
- [Zero-Shot Fact-Checking with Semantic Triples and Knowledge Graphs](https://mlsanity.com/paper/2312.11785/) (2023)
- [AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task](https://mlsanity.com/paper/2410.11446/) (2024)

## Built on by (most cited first, 2 of 2)

- [Needle in the Web: A Benchmark for Retrieving Targeted Web Pages in the Wild](https://mlsanity.com/paper/2512.16553/) (2025)
- [SOVER: Formal Certification of Optimization Reformulations via LLM-Assisted SMT Verification](https://mlsanity.com/paper/2609.00728/) (2026)

## Abstract (arXiv)

In this paper, we evaluate the ability of Large Language Models (LLMs) to assess the veracity of claims in ''news reports'' generated by themselves or other LLMs. Our goal is to determine whether LLMs can effectively fact-check their own content, using methods similar to those used to verify claims made by humans. Our findings indicate that LLMs are more effective at assessing claims in national or international news stories than in local news stories, better at evaluating static information than dynamic information, and better at verifying true claims compared to false ones. We hypothesize that this disparity arises because the former types of claims are better represented in the training data. Additionally, we find that incorporating retrieved results from a search engine in a Retrieval-Augmented Generation (RAG) setting significantly reduces the number of claims an LLM cannot assess. However, this approach also increases the occurrence of incorrect assessments, partly due to irrelevant or low-quality search results. This diagnostic study highlights the need for future research on fact-checking machine-generated reports to prioritize improving the precision and relevance of retrieved information to better support fact-checking efforts. Furthermore, claims about dynamic events and local news may require human-in-the-loop fact-checking systems to ensure accuracy and reliability.