# Identifying the sources of ideological bias in GPT models through linguistic variation in output

arXiv: https://arxiv.org/abs/2409.06043 · HTML: https://mlsanity.com/paper/2409.06043/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 44 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 8 of 8)

- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Gender bias and stereotypes in Large Language Models](https://mlsanity.com/paper/2308.14921/) (2023)
- [Large Language Models Can Be Used to Estimate the Latent Positions of Politicians](https://mlsanity.com/paper/2303.12057/) (2023)
- [Measurement in the Age of LLMs: An Application to Ideological Scaling](https://mlsanity.com/paper/2312.09203/) (2023)
- [Whose Side Are You On? Investigating the Political Stance of Large Language Models](https://mlsanity.com/paper/2403.13840/) (2024)
- [Assessing the Effectiveness of GPT-3 in Detecting False Political Statements: A Case Study on the LIAR Dataset](https://mlsanity.com/paper/2306.08190/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Cross-Lingual Analysis of Bias in Large Language Models Using Romanian History](https://mlsanity.com/paper/2510.02362/) (2025)

## Abstract (arXiv)

Extant work shows that generative AI models such as GPT-3.5 and 4 perpetuate social stereotypes and biases. One concerning but less explored source of bias is ideology. Do GPT models take ideological stances on politically sensitive topics? In this article, we provide an original approach to identifying ideological bias in generative models, showing that bias can stem from both the training data and the filtering algorithm. We leverage linguistic variation in countries with contrasting political attitudes to evaluate bias in average GPT responses to sensitive political topics in those languages. First, we find that GPT output is more conservative in languages that map well onto conservative societies (i.e., Polish), and more liberal in languages used uniquely in liberal societies (i.e., Swedish). This result provides strong evidence of training data bias in GPT models. Second, differences across languages observed in GPT-3.5 persist in GPT-4, even though GPT-4 is significantly more liberal due to OpenAI's filtering policy. Our main takeaway is that generative model training must focus on high-quality, curated datasets to reduce bias, even if it entails a compromise in training data size. Filtering responses after training only introduces new biases and does not remove the underlying training biases.