# Large Language Models' Detection of Political Orientation in Newspapers

arXiv: https://arxiv.org/abs/2406.00018 · HTML: https://mlsanity.com/paper/2406.00018/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 65 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 14 of 14)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Dbias: Detecting biases and ensuring Fairness in news articles](https://mlsanity.com/paper/2208.05777/) (2022)
- [Enabling News Consumers to View and Understand Biased News Coverage: A Study on the Perception and Visualization of Media Bias](https://mlsanity.com/paper/2105.09640/) (2021)
- [Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals](https://mlsanity.com/paper/2309.07601/) (2023)
- [Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models](https://mlsanity.com/paper/2310.11113/) (2023)
- [ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis](https://mlsanity.com/paper/2402.01715/) (2024)
- [The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias](https://mlsanity.com/paper/2312.16148/) (2023)
- [Exploiting Transformer-based Multitask Learning for the Detection of Media Bias in News Articles](https://mlsanity.com/paper/2211.03491/) (2022)
- [Text Analysis Using Deep Neural Networks in Digital Humanities and Information Science](https://mlsanity.com/paper/2307.16217/) (2023)
- [Exploring the Potential of the Large Language Models (LLMs) in Identifying Misleading News Headlines](https://mlsanity.com/paper/2405.03153/) (2024)

## Built on by (most cited first, 5 of 5)

- [FAIRGAME: a Framework for AI Agents Bias Recognition using Game Theory](https://mlsanity.com/paper/2504.14325/) (2025)
- [Media and responsible AI governance: a game-theoretic and LLM analysis](https://mlsanity.com/paper/2503.09858/) (2025)
- [Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents](https://mlsanity.com/paper/2504.08640/) (2025)
- [Can LLMs effectively provide game-theoretic-based scenarios for cybersecurity?](https://mlsanity.com/paper/2508.05670/) (2025)
- [When Numbers Start Talking: Implicit Numerical Coordination Among LLM-Based Agents](https://mlsanity.com/paper/2601.03846/) (2026)

## Abstract (arXiv)

Democratic opinion-forming may be manipulated if newspapers' alignment to political or economical orientation is ambiguous. Various methods have been developed to better understand newspapers' positioning. Recently, the advent of Large Language Models (LLM), and particularly the pre-trained LLM chatbots like ChatGPT or Gemini, hold disruptive potential to assist researchers and citizens alike. However, little is know on whether LLM assessment is trustworthy: do single LLM agrees with experts' assessment, and do different LLMs answer consistently with one another? In this paper, we address specifically the second challenge. We compare how four widely employed LLMs rate the positioning of newspapers, and compare if their answers align with one another. We observe that this is not the case. Over a woldwide dataset, articles in newspapers are positioned strikingly differently by single LLMs, hinting to inconsistent training or excessive randomness in the algorithms. We thus raise a warning when deciding which tools to use, and we call for better training and algorithm development, to cover such significant gap in a highly sensitive matter for democracy and societies worldwide. We also call for community engagement in benchmark evaluation, through our open initiative navai.pro.