# Are LLMs (Really) Ideological? An IRT-based Analysis and Alignment Tool for Perceived Socio-Economic Bias in LLMs

arXiv: https://arxiv.org/abs/2503.13149 · HTML: https://mlsanity.com/paper/2503.13149/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 140 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Builds on (most cited first, 8 of 8)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [The Political Preferences of LLMs](https://mlsanity.com/paper/2402.01789/) (2024)
- [How Susceptible are Large Language Models to Ideological Manipulation?](https://mlsanity.com/paper/2402.11725/) (2024)
- [Measuring Political Preferences in AI Systems: An Integrative Approach](https://mlsanity.com/paper/2503.10649/) (2025)
- [Choosing Between an LLM versus Search for Learning: A HigherEd Student Perspective](https://mlsanity.com/paper/2409.13051/) (2024)

## Built on by (most cited first, 1 of 1)

- [Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains](https://mlsanity.com/paper/2601.13137/) (2026)

## Abstract (arXiv)

We introduce an Item Response Theory (IRT)-based framework to detect and quantify socioeconomic bias in large language models (LLMs) without relying on subjective human judgments. Unlike traditional methods, IRT accounts for item difficulty, improving ideological bias estimation. We fine-tune two LLM families (Meta-LLaMa 3.2-1B-Instruct and Chat- GPT 3.5) to represent distinct ideological positions and introduce a two-stage approach: (1) modeling response avoidance and (2) estimating perceived bias in answered responses. Our results show that off-the-shelf LLMs often avoid ideological engagement rather than exhibit bias, challenging prior claims of partisanship. This empirically validated framework enhances AI alignment research and promotes fairer AI governance.