mlsanityData to 7 Oct 2026

Measuring Political Bias in Large Language Models: What Is Said and How It Is Said

arXiv 2403.18932 ↗First posted 27 Mar 2024cs.CL cs.AIcited by 89

Not on the map. Why?

Abstract (arXiv)

We propose to measure political bias in LLMs by analyzing both the content and style of their generated content regarding political issues. Existing benchmarks and measures focus on gender and racial biases. However, political bias exists in LLMs and can lead to polarization and other harms in downstream applications. In order to provide transparency to users, we advocate that there should be fine-grained and explainable measures of political biases generated by LLMs. Our proposed measure looks at different political issues such as reproductive rights and climate change, at both the content (the substance of the generation) and the style (the lexical polarity) of such bias. We measured the political bias in eleven open-sourced LLMs and showed that our proposed framework is easily scalable to other topics and is explainable.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 38 of its 75 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

38 of 75 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  9. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  10. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  11. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  12. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
26 more
  1. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  3. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  4. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  5. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  6. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  7. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  8. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  9. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  10. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  11. 2020We Can Detect Your Bias: Predicting the Political Ideology of News Articlescited by 92
  12. 2024The Political Preferences of LLMscited by 83
  13. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  14. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
  15. 2019In Plain Sight: Media Bias Through the Lens of Factual Reportingcited by 68
  16. 2019Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootingscited by 67
  17. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  18. 2022Neural Media Bias Detection Using Distant Supervision With BABE -- Bias Annotations By Expertscited by 54
  19. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 52
  20. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  21. 2020Analyzing Political Bias and Unfairness in News Articles at Different Levels of Granularitycited by 32
  22. 2020Weakly Supervised Learning of Nuanced Frames for Analyzing Polarization in News Mediacited by 27
  23. 2022NeuS: Neutral Multi-News Summarization for Mitigating Framing Biascited by 26
  24. 2021Assessing Political Prudence of Open-domain Chatbotscited by 20
  25. 2021To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violencecited by 12
  26. 2022No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Mediacited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 39; 2026: 35.

89 papers.

Sort:

By area:

  1. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  2. 2024On the Relationship between Truth and Political Bias in Language Modelscited by 31
  3. 2025Linear Representations of Political Perspective Emerge in Large Language Modelscited by 30
  4. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 22
  5. 2024AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessmentcited by 20
  6. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  7. 2024ComPO: Community Preferences for Language Model Personalizationcited by 19
  8. 2024Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personascited by 16
  9. 2025Unmasking Conversational Bias in AI Multiagent Systemscited by 16
  10. 2025Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Modelscited by 15
  11. 2024Designing AI-Agents with Personalities: A Psychometric Approachcited by 14
  12. 2025MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMscited by 13
38 more
  1. 2024Bias Amplification: Large Language Models as Increasingly Biased Mediacited by 12
  2. 2025LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are Highcited by 11
  3. 2025Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questionscited by 9
  4. 2024Bias in the Mirror: Are LLMs opinions robust to their own adversarial attacks ?cited by 8
  5. 2024Large Language Models Engineer Too Many Simple Features For Tabular Datacited by 8
  6. 2025Generative AI in Transportation Planning: A Surveycited by 8
  7. 2025Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classificationcited by 7
  8. 2024Aligning Large Language Models with Diverse Political Viewpointscited by 6
  9. 2025B-score: Detecting biases in large language models using response historycited by 6
  10. 2025Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Modelscited by 6
  11. 2024Revealing Fine-Grained Values and Opinions in Large Language Modelscited by 5
  12. 2025Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifierscited by 5
  13. 2025A Cautionary Tale About "Neutrally" Informative AI Tools Ahead of the 2025 Federal Elections in Germanycited by 5
  14. 2025Framing Political Bias in Multilingual LLMs Across Pakistani Languagescited by 5
  15. 2025Political Ideology Shifts in Large Language Modelscited by 5
  16. 2026Political Alignment in Large Language Models: A Multidimensional Audit of Psychometric Identity and Behavioral Biascited by 5
  17. 2026Replicating Human Motivated Reasoning Studies with LLMscited by 5
  18. 2026Positive Alignment: Artificial Intelligence for Human Flourishingcited by 5
  19. 2024PRISM: A Methodology for Auditing Biases in Large Language Modelscited by 4
  20. 2025Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Makingcited by 4
  21. 2026Jacobian Scopes: token-level causal attributions in LLMscited by 4
  22. 2025The Right to AIcited by 3
  23. 2025A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Modelscited by 3
  24. 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Modelscited by 3
  25. 2026PsychEthicsBench: Evaluating Large Language Models Against Australian Mental Health Ethicscited by 3
  26. 2026Architecting Trust in Artificial Epistemic Agentscited by 3
  27. 2024Balancing Rigor and Utility: Mitigating Cognitive Biases in Large Language Models for Multiple-Choice Questionscited by 2
  28. 2024MGM: Global Understanding of Audience Overlap Graphs for Predicting the Factuality and the Bias of News Mediacited by 2
  29. 2025Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speechcited by 2
  30. 2025Unraveling Interwoven Roles of Large Language Models in Authorship Privacy: Obfuscation, Mimicking, and Verificationcited by 2
  31. 2025Analysis of LLM Bias (Chinese Propaganda & Anti-US Sentiment) in DeepSeek-R1 vs. ChatGPT o3-mini-highcited by 2
  32. 2025Bias-Adjusted LLM Agents for Human-Like Decision-Making via Behavioral Economicscited by 2
  33. 2025A word association network methodology for evaluating implicit biases in LLMs compared to humanscited by 2
  34. 2025Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Searchcited by 2
  35. 2025Assessing instructor-AI cooperation for grading essay-type questions in an introductory sociology coursecited by 1
  36. 2025Exploring and Mitigating Fawning Hallucinations in Large Language Modelscited by 1
  37. 2025Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuningcited by 1
  38. 2025Don't Change My View: Ideological Bias Auditing in Large Language Modelscited by 1

50 most cited of 89 All 89 (JSON)