Are generative AI text annotations systematically biased?
Not on the map. Why?
Abstract (arXiv)
This paper investigates bias in GLLM annotations by conceptually replicating manual annotations of Boukes (2024). Using various GLLMs (Llama3.1:8b, Llama3.3:70b, GPT4o, Qwen2.5:72b) in combination with five different prompts for five concepts (political content, interactivity, rationality, incivility, and ideology). We find GLLMs perform adequate in terms of F1 scores, but differ from manual annotations in terms of prevalence, yield substantively different downstream results, and display systematic bias in that they overlap more with each other than with manual annotations. Differences in F1 scores fail to account for the degree of bias.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 7 of its 14 references that are in the map. First posted on arXiv 9 Dec 2025.
Builds on
7 of 14 references.
References · 7
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
- 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
- 2023Challenges and Applications of Large Language Modelscited by 251
- 2023Prediction-Powered Inferencecited by 214
- 2023Fairness And Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, And Mitigation Strategiescited by 83
- 2025Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotationcited by 43
- 2024Surprising gender biases in GPTcited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.