# From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference

arXiv: https://arxiv.org/abs/2609.24219 · HTML: https://mlsanity.com/paper/2609.24219/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models](https://mlsanity.com/paper/2304.00228/) (2023)
- [A Survey on Predicting the Factuality and the Bias of News Media](https://mlsanity.com/paper/2103.12506/) (2021)
- [Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals](https://mlsanity.com/paper/2309.07601/) (2023)
- [Reliability Estimation of News Media Sources: Birds of a Feather Flock Together](https://mlsanity.com/paper/2404.09565/) (2024)
- [Yesterday's News: Benchmarking Multi-Dimensional Out-of-Distribution Generalization of Misinformation Detection Models](https://mlsanity.com/paper/2410.18122/) (2024)
- [A Survey on Automatic Credibility Assessment Using Textual Credibility Signals in the Era of Large Language Models](https://mlsanity.com/paper/2410.21360/) (2024)
- [A Structured Analysis of Journalistic Evaluations for News Source Reliability](https://mlsanity.com/paper/2205.02736/) (2022)
- [Bridging Social Media and Search Engines: Dredge Words and the Detection of Unreliable Domains](https://mlsanity.com/paper/2406.11423/) (2024)
- [TROPIC - Trustworthiness Rating of Online Publishers through online Interactions Calculation](https://mlsanity.com/paper/2501.13561/) (2025)

## Abstract (arXiv)

Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual standards at source. When this process is translated into a computational approach, the problem is often formulated at the level of individual articles, with models being trained on a set of pre-labelled articles and their performance being evaluated in a test phase. In this work, we investigate news source reliability inference as a source-level prediction problem. We propose a two-stage framework in which transformer-based language models first estimate the reliability of individual articles and subsequently aggregate article-level predictions to infer the reliability of previously unseen publishers. To approximate realistic deployment conditions, we enforce a strict publisher-disjoint evaluation protocol, ensuring that no publisher appears in both training and test sets. Experiments on 19,476 political news articles from 439 English-language publishers labeled with NewsGuard reliability ratings show that aggregation substantially improves robustness and performance, increasing accuracy from approximately 0.60 at the article level to 0.69 at the publisher level. Finally, we analyze how prediction errors vary across political orientations, revealing statistically significant associations between political leaning and misclassification patterns. Overall, our findings show that publisher reliability can be inferred from aggregated textual signals alone, supporting scalable and content-based approaches to automated news source assessment.