# Beyond the Link: Assessing LLMs' ability to Classify Political Content across Global Media

arXiv: https://arxiv.org/abs/2506.17435 · HTML: https://mlsanity.com/paper/2506.17435/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Best Practices for Text Annotation with Large Language Models](https://mlsanity.com/paper/2402.05129/) (2024)
- [Positioning Political Texts with Large Language Models by Asking and Averaging](https://mlsanity.com/paper/2311.16639/) (2023)
- [Continuous Multi-Task Pre-training for Malicious URL Detection and Webpage Classification](https://mlsanity.com/paper/2402.11495/) (2024)
- [Characterizing Information Diets of Social Media Users](https://mlsanity.com/paper/1704.01442/) (2017)
- [Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Data](https://mlsanity.com/paper/2407.16516/) (2024)
- [Bias in LLMs as Annotators: The Effect of Party Cues on Labelling Decision by Large Language Models](https://mlsanity.com/paper/2408.15895/) (2024)

## Abstract (arXiv)

The use of large language models (LLMs) is becoming common in political science and digital media research. While LLMs have demonstrated ability in labelling tasks, their effectiveness to classify Political Content (PC) from URLs remains underexplored. This article evaluates whether LLMs can accurately distinguish PC from non-PC using both the text and the URLs of news articles across five countries (France, Germany, Spain, the UK, and the US) and their different languages. Using cutting-edge models, we benchmark their performance against human-coded data to assess whether URL-level analysis can approximate full-text analysis. Our findings show that URLs embed relevant information and can serve as a scalable, cost-effective alternative to discern PC. However, we also uncover systematic biases: LLMs seem to overclassify centrist news as political, leading to false positives that may distort further analyses. We conclude by outlining methodological recommendations on the use of LLMs in political science research.