# Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception

arXiv: https://arxiv.org/abs/2403.14896 · HTML: https://mlsanity.com/paper/2403.14896/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2024.

## Built on by (most cited first, 30 of 56)

- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions](https://mlsanity.com/paper/2409.16430/) (2024)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks](https://mlsanity.com/paper/2502.11090/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [BiasDora: Exploring Hidden Biased Associations in Vision-Language Models](https://mlsanity.com/paper/2407.02066/) (2024)
- [AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment](https://mlsanity.com/paper/2509.11620/) (2025)
- [Quantifying Generative Media Bias with a Corpus of Real-world and Generated News Articles](https://mlsanity.com/paper/2406.10773/) (2024)
- [Don't Get Too Excited -- Eliciting Emotions in LLMs](https://mlsanity.com/paper/2503.02457/) (2025)
- [Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2408.04643/) (2024)
- [Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMs](https://mlsanity.com/paper/2506.11415/) (2025)
- [Quantifying Trust: Financial Risk Management for Trustworthy AI Agents](https://mlsanity.com/paper/2604.03976/) (2026)
- [Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models](https://mlsanity.com/paper/2506.06751/) (2025)
- [Post-hoc Study of Climate Microtargeting on Social Media Ads with LLMs: Thematic Insights and Fairness Evaluation](https://mlsanity.com/paper/2410.05401/) (2024)
- [Neutralizing the Narrative: AI-Powered Debiasing of Online News Articles](https://mlsanity.com/paper/2504.03520/) (2025)
- [BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models](https://mlsanity.com/paper/2504.21299/) (2025)
- [ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs](https://mlsanity.com/paper/2506.01386/) (2025)
- [Who Gets Which Message? Auditing Demographic Bias in LLM-Generated Targeted Text](https://mlsanity.com/paper/2601.17172/) (2026)
- [Bias after Prompting: Persistent Discrimination in Large Language Models](https://mlsanity.com/paper/2509.08146/) (2025)
- [Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Models](https://mlsanity.com/paper/2602.22475/) (2026)
- [How Privacy-Savvy Are Large Language Models? A Case Study on Compliance and Privacy Technical Review](https://mlsanity.com/paper/2409.02375/) (2024)
- [Relative Bias: A Comparative Framework for Quantifying Bias in LLMs](https://mlsanity.com/paper/2505.17131/) (2025)
- [Multi-agent Systems for Misinformation Lifecycle : Detection, Correction And Source Identification](https://mlsanity.com/paper/2505.17511/) (2025)
- [How Similar Are Grokipedia and Wikipedia? A Multi-Dimensional Textual and Structural Comparison](https://mlsanity.com/paper/2510.26899/) (2025)
- [CLaRE-ty Amid Chaos: Quantifying Representational Entanglement to Predict Ripple Effects in LLM Editing](https://mlsanity.com/paper/2603.19297/) (2026)
- [Towards Safer Online Spaces: Simulating and Assessing Intervention Strategies for Eating Disorder Discussions](https://mlsanity.com/paper/2409.04043/) (2024)
- [Bias Similarity Measurement: A Black-Box Audit of Fairness Across LLMs](https://mlsanity.com/paper/2410.12010/) (2024)
- [Who Speaks Matters: Analysing the Influence of the Speaker's Ethnicity on Hate Classification](https://mlsanity.com/paper/2410.20490/) (2024)
- [Ideology-Based LLMs for Content Moderation](https://mlsanity.com/paper/2510.25805/) (2025)

## Abstract (arXiv)

The pervasive spread of misinformation and disinformation in social media underscores the critical importance of detecting media bias. While robust Large Language Models (LLMs) have emerged as foundational tools for bias prediction, concerns about inherent biases within these models persist. In this work, we investigate the presence and nature of bias within LLMs and its consequential impact on media bias detection. Departing from conventional approaches that focus solely on bias detection in media content, we delve into biases within the LLM systems themselves. Through meticulous examination, we probe whether LLMs exhibit biases, particularly in political bias prediction and text continuation tasks. Additionally, we explore bias across diverse topics, aiming to uncover nuanced variations in bias expression within the LLM framework. Importantly, we propose debiasing strategies, including prompt engineering and model fine-tuning. Extensive analysis of bias tendencies across different LLMs sheds light on the broader landscape of bias propagation in language models. This study advances our understanding of LLM bias, offering critical insights into its implications for bias detection tasks and paving the way for more robust and equitable AI systems