# Decoding News Narratives: A Critical Analysis of Large Language Models in Framing Detection

arXiv: https://arxiv.org/abs/2402.11621 · HTML: https://mlsanity.com/paper/2402.11621/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 36 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)
- [Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategies](https://mlsanity.com/paper/1808.09386/) (2018)
- [Modeling Framing in Immigration Discourse on Social Media](https://mlsanity.com/paper/2104.06443/) (2021)
- [DELPHI: Data for Evaluating LLMs' Performance in Handling Controversial Issues](https://mlsanity.com/paper/2310.18130/) (2023)

## Built on by (most cited first, 6 of 6)

- [Entity Framing and Role Portrayal in the News](https://mlsanity.com/paper/2502.14718/) (2025)
- [Frame In, Frame Out: Measuring Framing Bias in LLM-Generated News Summaries](https://mlsanity.com/paper/2505.05406/) (2025)
- [Left, Right, or Center? Evaluating LLM Framing in News Classification and Generation](https://mlsanity.com/paper/2601.05835/) (2026)
- [Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Markets](https://mlsanity.com/paper/2503.16674/) (2025)
- [Listening Between the Lines: Decoding Podcast Narratives with Language Modeling](https://mlsanity.com/paper/2511.05310/) (2025)
- [Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing](https://mlsanity.com/paper/2609.11769/) (2026)

## Abstract (arXiv)

The growing complexity and diversity of news coverage have made framing analysis a crucial yet challenging task in computational social science. Traditional approaches, including manual annotation and fine-tuned models, remain limited by high annotation costs, domain specificity, and inconsistent generalisation. Instruction-based large language models (LLMs) offer a promising alternative, yet their reliability for framing analysis remains insufficiently understood. In this paper, we conduct a systematic evaluation of several LLMs, including GPT-3.5/4, FLAN-T5, and Llama 3, across zero-shot, few-shot, and explanation-based prompting settings. Focusing on domain shift and inherent annotation ambiguity, we show that model performance is highly sensitive to prompt design and prone to systematic errors on ambiguous cases. Although LLMs, particularly GPT-4, exhibit stronger cross-domain generalisation, they also display systematic biases, most notably a tendency to conflate emotional language with framing. To enable principled evaluation under real-world topic diversity, we introduce a new dataset of out-of-domain news headlines covering diverse subjects. Finally, by analysing agreement patterns across multiple models on existing framing datasets, we demonstrate that cross-model consensus provides a useful signal for identifying contested annotations, offering a practical approach to dataset auditing in low-resource settings.