# IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance

arXiv: https://arxiv.org/abs/2502.08395 · HTML: https://mlsanity.com/paper/2502.08395/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Systematic Biases in LLM Simulations of Debates](https://mlsanity.com/paper/2402.04049/) (2024)
- [Measuring Political Bias in Large Language Models: What Is Said and How It Is Said](https://mlsanity.com/paper/2403.18932/) (2024)
- [The Political Preferences of LLMs](https://mlsanity.com/paper/2402.01789/) (2024)
- [Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters](https://mlsanity.com/paper/2410.24190/) (2024)
- [The Self-Perception and Political Biases of ChatGPT](https://mlsanity.com/paper/2304.07333/) (2023)
- [Are Large Language Models Consistent over Value-laden Questions?](https://mlsanity.com/paper/2407.02996/) (2024)
- [Large Language Models Reflect the Ideology of their Creators](https://mlsanity.com/paper/2410.18417/) (2024)
- [The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions](https://mlsanity.com/paper/2310.12418/) (2023)
- [Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation](https://mlsanity.com/paper/2402.12649/) (2024)
- [GlotLID: Language Identification for Low-Resource Languages](https://mlsanity.com/paper/2310.16248/) (2023)
- [Benchmarks as Microscopes: A Call for Model Metrology](https://mlsanity.com/paper/2407.16711/) (2024)
- [Assessing Political Bias in Large Language Models](https://mlsanity.com/paper/2405.13041/) (2024)
- [On the Relationship between Truth and Political Bias in Language Models](https://mlsanity.com/paper/2409.05283/) (2024)
- [How Susceptible are Large Language Models to Ideological Manipulation?](https://mlsanity.com/paper/2402.11725/) (2024)
- [Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks](https://mlsanity.com/paper/2112.07475/) (2021)
- [Quantifying Generative Media Bias with a Corpus of Real-world and Generated News Articles](https://mlsanity.com/paper/2406.10773/) (2024)

## Built on by (most cited first, 26 of 26)

- [Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.16534/) (2025)
- [Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models](https://mlsanity.com/paper/2503.16148/) (2025)
- [Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race](https://mlsanity.com/paper/2506.00253/) (2025)
- [What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models](https://mlsanity.com/paper/2510.04226/) (2025)
- [What Large Language Models Do Not Talk About: An Empirical Study of Moderation and Censorship Practices](https://mlsanity.com/paper/2504.03803/) (2025)
- [One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization](https://mlsanity.com/paper/2601.18572/) (2026)
- [What Is The Political Content in LLMs' Pre- and Post-Training Data?](https://mlsanity.com/paper/2509.22367/) (2025)
- [Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMs](https://mlsanity.com/paper/2508.05553/) (2025)
- [Political Ideology Shifts in Large Language Models](https://mlsanity.com/paper/2508.16013/) (2025)
- [Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias](https://mlsanity.com/paper/2601.18486/) (2026)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [MillStone: How Open-Minded Are LLMs?](https://mlsanity.com/paper/2509.11967/) (2025)
- [DIVERGE: Diversity-Enhanced RAG for Open-Ended Information Seeking](https://mlsanity.com/paper/2602.00238/) (2026)
- [DeliberationBench: A Normative Benchmark for the Influence of Large Language Models on Users' Views](https://mlsanity.com/paper/2603.10018/) (2026)
- [How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions](https://mlsanity.com/paper/2604.22503/) (2026)
- [Argument Collapse: LLMs Flatten Long-Form Public Debate](https://mlsanity.com/paper/2606.01736/) (2026)
- [Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset](https://mlsanity.com/paper/2510.20508/) (2025)
- [Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners](https://mlsanity.com/paper/2511.05501/) (2025)
- [How Value Induction Reshapes LLM Behaviour](https://mlsanity.com/paper/2605.07925/) (2026)
- [Large Language Models are Perplexed by some Political Parties](https://mlsanity.com/paper/2606.05937/) (2026)
- [AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Models](https://mlsanity.com/paper/2606.16127/) (2026)
- [Issue Bias in Generative AI Writing Assistance: Political Issues and LLMs in the Swedish 2026 Election](https://mlsanity.com/paper/2609.15207/) (2026)
- [Narrowing the Horizon: Quantifying Topic Saliency Shifts in Generative Monoculture](https://mlsanity.com/paper/2609.35302/) (2026)
- [Stance Drift: How AI-mediated Communication Distorts Our Message](https://mlsanity.com/paper/2610.04620/) (2026)
- [The AI's Poker Face: Contrasting Persuasion and Sycophancy in Voice-Based Multi-Turn Conversations](https://mlsanity.com/paper/2610.14124/) (2026)
- [Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively](https://mlsanity.com/paper/2610.17703/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns about issue bias, where an LLM tends to present just one perspective on a given issue, which in turn may influence how users think about this issue. So far, it has not been possible to measure which issue biases LLMs manifest in real user interactions, making it difficult to address the risks from biased LLMs. Therefore, we create IssueBench: a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance, which we construct based on 3.9k templates (e.g. "write a blog about") and 212 political issues (e.g. "AI regulation") from real user interactions. Using IssueBench, we show that issue biases are common and persistent in 10 state-of-the-art LLMs. We also show that biases are very similar across models, and that all models align more with US Democrat than Republican voter opinion on a subset of issues. IssueBench can easily be adapted to include other issues, templates, or tasks. By enabling robust and realistic measurement, we hope that IssueBench can bring a new quality of evidence to ongoing discussions about LLM biases and how to address them.