# Generating Media Background Checks for Automated Source Critical Reasoning

arXiv: https://arxiv.org/abs/2409.00781 · HTML: https://mlsanity.com/paper/2409.00781/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 55 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 19 of 19)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [What Evidence Do Language Models Find Convincing?](https://mlsanity.com/paper/2402.11782/) (2024)
- [Predicting Factuality of Reporting and Bias of News Media Sources](https://mlsanity.com/paper/1810.01765/) (2018)
- [What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context](https://mlsanity.com/paper/2005.04518/) (2020)
- [Early Detection of Fake News by Utilizing the Credibility of News, Publishers, and Users Based on Weakly Supervised Learning](https://mlsanity.com/paper/2012.04233/) (2020)
- [The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who](https://mlsanity.com/paper/2304.14238/) (2023)
- [CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding](https://mlsanity.com/paper/2105.10912/) (2021)
- [Tanbih: Get To Know What You Are Reading](https://mlsanity.com/paper/1910.02028/) (2019)
- [Standardize: Aligning Language Models with Expert-Defined Standards for Content Generation](https://mlsanity.com/paper/2402.12593/) (2024)

## Built on by (most cited first, 10 of 10)

- [Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking](https://mlsanity.com/paper/2502.09083/) (2025)
- [To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts](https://mlsanity.com/paper/2410.14675/) (2024)
- [Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs](https://mlsanity.com/paper/2505.17762/) (2025)
- [COVE: COntext and VEracity prediction for out-of-context images](https://mlsanity.com/paper/2502.01194/) (2025)
- [Attacks by Content: Automated Fact-checking is an AI Security Issue](https://mlsanity.com/paper/2510.11238/) (2025)
- [MGM: Global Understanding of Audience Overlap Graphs for Predicting the Factuality and the Bias of News Media](https://mlsanity.com/paper/2412.10467/) (2024)
- [AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.25092/) (2026)
- [MEDIAREF: A Public Knowledge Store for Media Background Checks](https://mlsanity.com/paper/2607.02383/) (2026)
- [Lazy Grounding: Attacking Search Agents with Factual Evidence](https://mlsanity.com/paper/2608.30303/) (2026)
- [OpenFC: Learning Verification Policies towards Open-Search Fact Checking](https://mlsanity.com/paper/2609.33579/) (2026)

## Abstract (arXiv)

Not everything on the internet is true. This unfortunate fact requires both humans and models to perform complex reasoning about credibility when working with retrieved information. In NLP, this problem has seen little attention. Indeed, retrieval-augmented models are not typically expected to distrust retrieved documents. Human experts overcome the challenge by gathering signals about the context, reliability, and tendency of source documents - that is, they perform source criticism. We propose a novel NLP task focused on finding and summarising such signals. We introduce a new dataset of 6,709 "media background checks" derived from Media Bias / Fact Check, a volunteer-run website documenting media bias. We test open-source and closed-source LLM baselines with and without retrieval on this dataset, finding that retrieval greatly improves performance. We furthermore carry out human evaluation, demonstrating that 1) media background checks are helpful for humans, and 2) media background checks are helpful for retrieval-augmented models.