# Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts

arXiv: https://arxiv.org/abs/2506.12552 · HTML: https://mlsanity.com/paper/2506.12552/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 14 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [A Stylometric Inquiry into Hyperpartisan and Fake News](https://mlsanity.com/paper/1702.05638/) (2017)
- [Predicting Factuality of Reporting and Bias of News Media Sources](https://mlsanity.com/paper/1810.01765/) (2018)
- [What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context](https://mlsanity.com/paper/2005.04518/) (2020)
- [Multi-view Models for Political Ideology Detection of News Articles](https://mlsanity.com/paper/1809.03485/) (2018)
- [Multi-Task Ordinal Regression for Jointly Predicting the Trustworthiness and the Leading Political Ideology of News Media](https://mlsanity.com/paper/1904.00542/) (2019)
- [Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models](https://mlsanity.com/paper/2304.00228/) (2023)
- [Knowledge-Based Trust: Estimating the Trustworthiness of Web Sources](https://mlsanity.com/paper/1502.03519/) (2015)
- [A Survey on Predicting the Factuality and the Bias of News Media](https://mlsanity.com/paper/2103.12506/) (2021)
- [GREENER: Graph Neural Networks for News Media Profiling](https://mlsanity.com/paper/2211.05533/) (2022)
- [An Interactive Framework for Profiling News Media Sources](https://mlsanity.com/paper/2309.07384/) (2023)
- [Team QCRI-MIT at SemEval-2019 Task 4: Propaganda Analysis Meets Hyperpartisan News Detection](https://mlsanity.com/paper/1904.03513/) (2019)
- [MGM: Global Understanding of Audience Overlap Graphs for Predicting the Factuality and the Bias of News Media](https://mlsanity.com/paper/2412.10467/) (2024)

## Built on by (most cited first, 3 of 3)

- [Political Alignment in Large Language Models: A Multidimensional Audit of Psychometric Identity and Behavioral Bias](https://mlsanity.com/paper/2601.06194/) (2026)
- [Assessing Web Search Credibility and Response Groundedness in Chat Assistants](https://mlsanity.com/paper/2510.13749/) (2025)
- [Can temporal article-level credibility signals improve domain-level credibility prediction?](https://mlsanity.com/paper/2607.04560/) (2026)

## Abstract (arXiv)

In an age characterized by the proliferation of mis- and disinformation online, it is critical to empower readers to understand the content they are reading. Important efforts in this direction rely on manual or automatic fact-checking, which can be challenging for emerging claims with limited information. Such scenarios can be handled by assessing the reliability and the political bias of the source of the claim, i.e., characterizing entire news outlets rather than individual claims or articles. This is an important but understudied research direction. While prior work has looked into linguistic and social contexts, we do not analyze individual articles or information in social media. Instead, we propose a novel methodology that emulates the criteria that professional fact-checkers use to assess the factuality and political bias of an entire outlet. Specifically, we design a variety of prompts based on these criteria and elicit responses from large language models (LLMs), which we aggregate to make predictions. In addition to demonstrating sizable improvements over strong baselines via extensive experiments with multiple LLMs, we provide an in-depth error analysis of the effect of media popularity and region on model performance. Further, we conduct an ablation study to highlight the key components of our dataset that contribute to these improvements. To facilitate future research, we released our dataset and code at https://github.com/mbzuai-nlp/llm-media-profiling.