mlsanityData to 7 Oct 2026

On the Relationship between Truth and Political Bias in Language Models

arXiv 2409.05283 ↗First posted 9 Sep 2024cs.CL cs.AIcited by 31

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Language model alignment research often attempts to ensure that models are not only helpful and harmless, but also truthful and unbiased. However, optimizing these objectives simultaneously can obscure how improving one aspect might impact the others. In this work, we focus on analyzing the relationship between two concepts essential in both language model alignment and political science: truthfulness and political bias. We train reward models on various popular truthfulness datasets and subsequently evaluate their political bias. Our findings reveal that optimizing reward models for truthfulness on these datasets tends to result in a left-leaning political bias. We also find that existing open-source reward models (i.e., those trained on standard human preference datasets) already show a similar bias and that the bias is larger for larger models. These results raise important questions about the datasets used to represent truthfulness, potential limitations of aligning models to be both truthful and politically unbiased, and what language models capture about the relationship between truth and politics.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 44 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

23 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  2. 2020Learning to summarize from human feedback2 Sep

Other references · 21

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2023Whose Opinions Do Language Models Reflect?cited by 759
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  10. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  12. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
9 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  3. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  4. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  5. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  6. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  7. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 77
  8. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
  9. 2023Challenges with unsupervised LLM knowledge discoverycited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 13.

31 papers.

Sort:

By area:

  1. 2025SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Modelscited by 28
  2. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 22
  3. 2025Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questionscited by 9
  4. 2025IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignmentcited by 8
  5. 2025What Is The Political Content in LLMs' Pre- and Post-Training Data?cited by 7
  6. 2025Benchmarking Overton Pluralism in LLMscited by 7
  7. 2025Mitigating Bias in RAG: Controlling the Embeddercited by 6
  8. 2025Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language modelscited by 5
  9. 2025AccessEval: Benchmarking Disability Bias in Large Language Modelscited by 5
  10. 2025Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignmentcited by 4
  11. 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Modelscited by 3
  12. 2026Reward Models Inherit Value Biases from Pretrainingcited by 3
19 more
  1. 2025Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Searchcited by 2
  2. 2026PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplaycited by 2
  3. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  4. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?cited by 1
  5. 2025From Delegates to Trustees: How Optimizing for Long-Term Interests Shapes Bias and Alignment in LLMcited by 1
  6. 2025Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncodercited by 0
  7. 2025Counterspeech for Mitigating the Influence of Media Bias: Comparing Human and LLM-Generated Responsescited by 0
  8. 2025Who Gets Left Behind? Auditing Disability Inclusivity in Large Language Modelscited by 0
  9. 2025Measuring Algorithmic Partisanship via Zero-Shot Classification and Its Implications on Political Discoursecited by 0
  10. 2026Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitationcited by 0
  11. 2026Fairness-Aware Retrieval Optimization for Retrieval-Augmented Generationcited by 0
  12. 2026Reducing Political Manipulation with Consistency Trainingcited by 0
  13. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biasescited by 0
  14. 2026SAE-StatSteer: Statistical Consensus Feature Selection for Optimization-Free Activation Steering of Large Language Modelscited by 0
  15. 2026Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimescited by 0
  16. 2026Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignmentcited by 0
  17. 2026IDEEA: training-free Input-Dependent stEEring via Activation cluster matchingcited by 0
  18. 2026Issue Bias in Generative AI Writing Assistance: Political Issues and LLMs in the Swedish 2026 Electioncited by 0
  19. 2026When Steering Fails in Latent Reasoning: A Latent-to-Language Transition Gapcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Political Alignment in Large Language Models: A Multidimensional Audit of Psychometric Identity and Behavioral Bias
  2. 2025Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes
  3. 2024PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models
  4. 2025Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMs
  5. 2025Measuring Political Preferences in AI Systems: An Integrative Approach
  6. 2026A Multi-Dimensional Audit of Politically Aligned Large Language Models
  7. 2024The Political Preferences of LLMs
  8. 2024Political Bias in LLMs: Unaligned Moral Values in Agent-centric Simulations