# Does Topic Sentiment Cause Perceived Ideology? Comparing Human and LLM Annotations in Political News Articles

arXiv: https://arxiv.org/abs/2606.06715 · HTML: https://mlsanity.com/paper/2606.06715/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation](https://mlsanity.com/paper/2003.03934/) (2020)
- [Explore Spurious Correlations at the Concept Level in Language Models for Text Classification](https://mlsanity.com/paper/2311.08648/) (2023)
- [Large Language Models Can Be Used to Estimate the Latent Positions of Politicians](https://mlsanity.com/paper/2303.12057/) (2023)
- [Evaluation Methods and Measures for Causal Learning Algorithms](https://mlsanity.com/paper/2202.02896/) (2022)
- [LLMs Are Prone to Fallacies in Causal Inference](https://mlsanity.com/paper/2406.12158/) (2024)
- [Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study](https://mlsanity.com/paper/2412.13169/) (2024)
- [Topic-Specific Sentiment Analysis Can Help Identify Political Ideology](https://mlsanity.com/paper/1810.12897/) (2018)
- [Text as Causal Mediators: Research Design for Causal Estimates of Differential Treatment of Social Groups via Language Aspects](https://mlsanity.com/paper/2109.07542/) (2021)
- [Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Ability](https://mlsanity.com/paper/2411.19456/) (2024)

## Abstract (arXiv)

We ask whether topic sentiment has a causal effect on perceived political ideology, and whether the answer depends on who assigns the ideology label. Using articles from AllSides, paired with shared sentiment annotations from Llama-3.3-70b-versatile, we compare ideology labels from expert human annotators, GPT-4o-mini (baseline and finetuned), and Llama-3.3-70B. We apply Double Machine Learning (DML) and mediation analysis across all four annotation paradigms. Zero-shot LLMs regularly inflate effect sizes relative to human annotations, while fine-tuning often attenuates them back toward the human scale. Our results have implications for the use of LLM annotations as silver labels and as proxies for human judgment in downstream causal analyses: they may be reliable for recovering the presence and direction of effects on the partisan topics, but not their magnitude, leading to over- or under-prediction of some ideology given particular topics.