mlsanityData to 14 Oct 2026

IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance

arXiv 2502.08395 ↗First posted 12 Feb 2025cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns about issue bias, where an LLM tends to present just one perspective on a given issue, which in turn may influence how users think about this issue. So far, it has not been possible to measure which issue biases LLMs manifest in real user interactions, making it difficult to address the risks from biased LLMs. Therefore, we create IssueBench: a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance, which we construct based on 3.9k templates (e.g. "write a blog about") and 212 political issues (e.g. "AI regulation") from real user interactions. Using IssueBench, we show that issue biases are common and persistent in 10 state-of-the-art LLMs. We also show that biases are very similar across models, and that all models align more with US Democrat than Republican voter opinion on a subset of issues. IssueBench can easily be adapted to include other issues, templates, or tasks. By enabling robust and realistic measurement, we hope that IssueBench can bring a new quality of evidence to ongoing discussions about LLM biases and how to address them.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 28 of its 55 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

28 of 55 references.

References · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023Whose Opinions Do Language Models Reflect?cited by 773
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  7. 20242 OLMo 2 Furiouscited by 340
  8. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  9. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  10. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  11. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  12. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
16 more
  1. 2024Systematic Biases in LLM Simulations of Debatescited by 99
  2. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  3. 2024The Political Preferences of LLMscited by 84
  4. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  5. 2023The Self-Perception and Political Biases of ChatGPTcited by 66
  6. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  7. 2024Large Language Models Reflect the Ideology of their Creatorscited by 57
  8. 2023The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactionscited by 45
  9. 2024Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluationcited by 41
  10. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  11. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  12. 2024Assessing Political Bias in Large Language Modelscited by 32
  13. 2024On the Relationship between Truth and Political Bias in Language Modelscited by 31
  14. 2024How Susceptible are Large Language Models to Ideological Manipulation?cited by 26
  15. 2021Two Contrasting Data Annotation Paradigms for Subjective NLP Taskscited by 12
  16. 2024Quantifying Generative Media Bias with a Corpus of Real-world and Generated News Articlescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 15.

26 papers.

Sort:

By area:

  1. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  2. 2025Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Modelscited by 15
  3. 2025Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Racecited by 15
  4. 2025What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Modelscited by 14
  5. 2025What Large Language Models Do Not Talk About: An Empirical Study of Moderation and Censorship Practicescited by 11
  6. 2026One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalizationcited by 9
  7. 2025What Is The Political Content in LLMs' Pre- and Post-Training Data?cited by 7
  8. 2025Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMscited by 6
  9. 2025Political Ideology Shifts in Large Language Modelscited by 5
  10. 2026Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Biascited by 4
  11. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  12. 2025MillStone: How Open-Minded Are LLMs?cited by 1
14 more
  1. 2026DIVERGE: Diversity-Enhanced RAG for Open-Ended Information Seekingcited by 1
  2. 2026DeliberationBench: A Normative Benchmark for the Influence of Large Language Models on Users' Viewscited by 1
  3. 2026How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinionscited by 1
  4. 2026Argument Collapse: LLMs Flatten Long-Form Public Debatecited by 1
  5. 2025Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Datasetcited by 0
  6. 2025Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitionerscited by 0
  7. 2026How Value Induction Reshapes LLM Behaviourcited by 0
  8. 2026Large Language Models are Perplexed by some Political Partiescited by 0
  9. 2026AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Modelscited by 0
  10. 2026Issue Bias in Generative AI Writing Assistance: Political Issues and LLMs in the Swedish 2026 Electioncited by 0
  11. 2026Narrowing the Horizon: Quantifying Topic Saliency Shifts in Generative Monoculturecited by 0
  12. 2026Stance Drift: How AI-mediated Communication Distorts Our Messagecited by 0
  13. 2026The AI's Poker Face: Contrasting Persuasion and Sycophancy in Voice-Based Multi-Turn Conversationscited by 0
  14. 2026Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectivelycited by 0