# LLMs left, right, and center: Assessing GPT's capabilities to label political bias from web domains

arXiv: https://arxiv.org/abs/2407.14344 · HTML: https://mlsanity.com/paper/2407.14344/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Builds on (most cited first, 10 of 10)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [Large Language Models Can Be Used to Estimate the Latent Positions of Politicians](https://mlsanity.com/paper/2303.12057/) (2023)
- [Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models](https://mlsanity.com/paper/2304.00228/) (2023)
- [The Importance of Human-Labeled Data in the Era of LLMs](https://mlsanity.com/paper/2306.14910/) (2023)
- [Evaluating Twitter's Algorithmic Amplification of Low-Credibility Content: An Observational Study](https://mlsanity.com/paper/2305.06125/) (2023)

## Built on by (most cited first, 9 of 9)

- [The simulation of judgment in LLMs](https://mlsanity.com/paper/2502.04426/) (2025)
- [Towards Operationalizing Right to Data Protection](https://mlsanity.com/paper/2411.08506/) (2024)
- [Decoding News Bias: Multi Bias Detection in News Articles](https://mlsanity.com/paper/2501.02482/) (2025)
- [Auditing Google's Search Algorithm: Measuring News Diversity Across Brazil, the UK, and the US](https://mlsanity.com/paper/2410.23842/) (2024)
- [Social media polarization during conflict: Insights from an ideological stance dataset on Israel-Palestine Reddit comments](https://mlsanity.com/paper/2502.00414/) (2025)
- [Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Markets](https://mlsanity.com/paper/2503.16674/) (2025)
- [A Multi-Dimensional Evaluation of Explainability in Media Bias Detection](https://mlsanity.com/paper/2607.19954/) (2026)
- [Asymmetric Discourse Homogenization and Shared Language Technology: Evidence from Reddit](https://mlsanity.com/paper/2608.13674/) (2026)
- [Ideological Stance Detection in a Low-Resource Language: Polarization in Bangladeshi Public vs Private University Discourse on Social Media](https://mlsanity.com/paper/2610.04401/) (2026)

## Abstract (arXiv)

This research investigates whether OpenAI's GPT-4, a state-of-the-art large language model, can accurately classify the political bias of news sources based solely on their URLs. Given the subjective nature of political labels, third-party bias ratings like those from Ad Fontes Media, AllSides, and Media Bias/Fact Check (MBFC) are often used in research to analyze news source diversity. This study aims to determine if GPT-4 can replicate these human ratings on a seven-degree scale ("far-left" to "far-right"). The analysis compares GPT-4's classifications against MBFC's, and controls for website popularity using Open PageRank scores. Findings reveal a high correlation (Spearman's  ρ = .89, n = 5,877, p < 0.001) between GPT-4's and MBFC's ratings, indicating the model's potential reliability. However, GPT-4 abstained from classifying approximately frac23 of the dataset. It is more likely to abstain from rating unpopular websites, which also suffer from less accurate assessments. The LLM tends to avoid classifying sources that MBFC considers to be centrist, resulting in more polarized outputs. Finally, this analysis shows a slight leftward skew in GPT's classifications compared to MBFC's. Therefore, while this paper suggests that while GPT-4 can be a scalable, cost-effective tool for political bias classification of news websites, its use should be as a complement to human judgment to mitigate biases.