# Large Language Model (LLM) Bias Index -- LLMBI

arXiv: https://arxiv.org/abs/2312.14769 · HTML: https://mlsanity.com/paper/2312.14769/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 18 references that are in the map.
- First posted on arXiv 22 Dec 2023.

## Builds on (most cited first, 7 of 7)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [AI Fairness for People with Disabilities: Point of View](https://mlsanity.com/paper/1811.10670/) (2018)

## Built on by (most cited first, 14 of 14)

- [NoveltyBench: Evaluating Language Models for Humanlike Diversity](https://mlsanity.com/paper/2504.05228/) (2025)
- [Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View](https://mlsanity.com/paper/2405.14744/) (2024)
- [A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions](https://mlsanity.com/paper/2409.16430/) (2024)
- [Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategies](https://mlsanity.com/paper/2501.08441/) (2025)
- [The Generation Gap: Exploring Age Bias in the Value Systems of Large Language Models](https://mlsanity.com/paper/2404.08760/) (2024)
- [AI Governance through Markets](https://mlsanity.com/paper/2501.17755/) (2025)
- [Broadening Access to Simulations for End-Users via Large Language Models: Challenges and Opportunities](https://mlsanity.com/paper/2409.15290/) (2024)
- [Mechanistic Interpretability with SAEs: Probing Religion, Violence, and Geography in Large Language Models](https://mlsanity.com/paper/2509.17665/) (2025)
- [Bias Similarity Measurement: A Black-Box Audit of Fairness Across LLMs](https://mlsanity.com/paper/2410.12010/) (2024)
- [Ethical Concerns of Generative AI and Mitigation Strategies: A Systematic Mapping Study](https://mlsanity.com/paper/2502.00015/) (2025)
- [Quantifying Gender Bias in Large Language Models: When ChatGPT Becomes a Hiring Manager](https://mlsanity.com/paper/2604.00011/) (2026)
- [Prompt Recursive Search: A Living Framework with Adaptive Growth in LLM Auto-Prompting](https://mlsanity.com/paper/2408.01423/) (2024)
- [CLEAR: A Comprehensive Linguistic Evaluation of Argument Rewriting by Large Language Models](https://mlsanity.com/paper/2509.15027/) (2025)
- [What AI Speaks for Your Community: Polling AI Agents for Public Opinion on Data Center Projects](https://mlsanity.com/paper/2511.22037/) (2025)

## Abstract (arXiv)

The Large Language Model Bias Index (LLMBI) is a pioneering approach designed to quantify and address biases inherent in large language models (LLMs), such as GPT-4. We recognise the increasing prevalence and impact of LLMs across diverse sectors. This research introduces a novel metric, LLMBI, to systematically measure and mitigate biases potentially skewing model responses. We formulated LLMBI using a composite scoring system incorporating multiple dimensions of bias, including but not limited to age, gender, and racial biases. To operationalise this metric, we engaged in a multi-step process involving collecting and annotating LLM responses, applying sophisticated Natural Language Processing (NLP) techniques for bias detection, and computing the LLMBI score through a specially crafted mathematical formula. The formula integrates weighted averages of various bias dimensions, a penalty for dataset diversity deficiencies, and a correction for sentiment biases. Our empirical analysis, conducted using responses from OpenAI's API, employs advanced sentiment analysis as a representative method for bias detection. The research reveals LLMs, whilst demonstrating impressive capabilities in text generation, exhibit varying degrees of bias across different dimensions. LLMBI provides a quantifiable measure to compare biases across models and over time, offering a vital tool for systems engineers, researchers and regulators in enhancing the fairness and reliability of LLMs. It highlights the potential of LLMs in mimicking unbiased human-like responses. Additionally, it underscores the necessity of continuously monitoring and recalibrating such models to align with evolving societal norms and ethical standards.