# Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models

arXiv: https://arxiv.org/abs/2304.00228 · HTML: https://mlsanity.com/paper/2304.00228/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 55 references that are in the map.
- First posted on arXiv 1 Apr 2023.

## Builds on (most cited first, 13 of 13)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity](https://mlsanity.com/paper/2209.12106/) (2022)
- [Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation](https://mlsanity.com/paper/1806.01156/) (2018)
- [Examining the consumption of radical content on YouTube](https://mlsanity.com/paper/2011.12843/) (2020)
- [Neutral bots probe political bias on social media](https://mlsanity.com/paper/2005.08141/) (2020)
- [Social media sharing by political elites: An asymmetric American exceptionalism](https://mlsanity.com/paper/2207.06313/) (2022)

## Built on by (most cited first, 25 of 25)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [ChatGPT Informed Graph Neural Network for Stock Movement Prediction](https://mlsanity.com/paper/2306.03763/) (2023)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks](https://mlsanity.com/paper/2404.16966/) (2024)
- [Can Large Language Models Understand Context?](https://mlsanity.com/paper/2402.00858/) (2024)
- [Embedding Democratic Values into Social Media AIs via Societal Objective Functions](https://mlsanity.com/paper/2307.13912/) (2023)
- [The simulation of judgment in LLMs](https://mlsanity.com/paper/2502.04426/) (2025)
- [User Modeling in the Era of Large Language Models: Current Research and Future Directions](https://mlsanity.com/paper/2312.11518/) (2023)
- [Whose Facts Win? LLM Source Preferences under Knowledge Conflicts](https://mlsanity.com/paper/2601.03746/) (2026)
- [LLMs left, right, and center: Assessing GPT's capabilities to label political bias from web domains](https://mlsanity.com/paper/2407.14344/) (2024)
- [Global-Liar: Factuality of LLMs over Time and Geographic Regions](https://mlsanity.com/paper/2401.17839/) (2024)
- [Source Coverage and Citation Bias in LLM-based vs. Traditional Search Engines](https://mlsanity.com/paper/2512.09483/) (2025)
- [Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future to eDemorcacy](https://mlsanity.com/paper/2504.00241/) (2025)
- [Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts](https://mlsanity.com/paper/2506.12552/) (2025)
- [Stress index strategy enhanced with financial news sentiment analysis for the equity markets](https://mlsanity.com/paper/2404.00012/) (2024)
- [How Similar Are Grokipedia and Wikipedia? A Multi-Dimensional Textual and Structural Comparison](https://mlsanity.com/paper/2510.26899/) (2025)
- [Framing Social Movements on Social Media: Unpacking Diagnostic, Prognostic, and Motivational Strategies](https://mlsanity.com/paper/2406.13820/) (2024)
- [MERIT: Modular Framework for Multimodal Misinformation Detection with Web-Grounded Reasoning](https://mlsanity.com/paper/2510.17590/) (2025)
- [Information Discernment in Large Language Models](https://mlsanity.com/paper/2607.19355/) (2026)
- [CrediBench: Building Web-Scale Network Datasets for Information Integrity](https://mlsanity.com/paper/2509.23340/) (2025)
- [RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty](https://mlsanity.com/paper/2602.12424/) (2026)
- [Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmark](https://mlsanity.com/paper/2603.17838/) (2026)
- [Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources](https://mlsanity.com/paper/2607.03365/) (2026)
- [From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference](https://mlsanity.com/paper/2609.24219/) (2026)

## Abstract (arXiv)

Search engines increasingly leverage large language models (LLMs) to generate direct answers, and AI chatbots now access the Internet for fresh data. As information curators for billions of users, LLMs must assess the accuracy and reliability of different sources. This paper audits nine widely used LLMs from three leading providers -- OpenAI, Google, and Meta -- to evaluate their ability to discern credible and high-quality information sources from low-credibility ones. We find that while LLMs can rate most tested news outlets, larger models more frequently refuse to provide ratings due to insufficient information, whereas smaller models are more prone to making errors in their ratings. For sources where ratings are provided, LLMs exhibit a high level of agreement among themselves (average Spearman's ρ = 0.79), but their ratings align only moderately with human expert evaluations (average ρ = 0.50). Analyzing news sources with different political leanings in the US, we observe a liberal bias in credibility ratings yielded by all LLMs in default configurations. Additionally, assigning partisan roles to LLMs consistently induces strong politically congruent bias in their ratings. These findings have important implications for the use of LLMs in curating news and political information.