# BEATS: Bias Evaluation and Assessment Test Suite for Large Language Models

arXiv: https://arxiv.org/abs/2503.24310 · HTML: https://mlsanity.com/paper/2503.24310/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2025.

## Built on by (most cited first, 5 of 5)

- [Bi-directional Bias Attribution: Debiasing Large Language Models without Modifying Prompts](https://mlsanity.com/paper/2602.04398/) (2026)
- [Data and AI governance: Promoting equity, ethics, and fairness in large language models](https://mlsanity.com/paper/2508.03970/) (2025)
- [Unified Threat Detection and Mitigation Framework (UTDMF): Combating Prompt Injection, Deception, and Bias in Enterprise-Scale Transformers](https://mlsanity.com/paper/2510.04528/) (2025)
- [Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations](https://mlsanity.com/paper/2512.00556/) (2025)
- [SHARP: Social Harm Analysis via Risk Profiles for Measuring Inequities in Large Language Models](https://mlsanity.com/paper/2601.21235/) (2026)

## Abstract (arXiv)

In this research, we introduce BEATS, a novel framework for evaluating Bias, Ethics, Fairness, and Factuality in Large Language Models (LLMs). Building upon the BEATS framework, we present a bias benchmark for LLMs that measure performance across 29 distinct metrics. These metrics span a broad range of characteristics, including demographic, cognitive, and social biases, as well as measures of ethical reasoning, group fairness, and factuality related misinformation risk. These metrics enable a quantitative assessment of the extent to which LLM generated responses may perpetuate societal prejudices that reinforce or expand systemic inequities. To achieve a high score on this benchmark a LLM must show very equitable behavior in their responses, making it a rigorous standard for responsible AI evaluation. Empirical results based on data from our experiment show that, 37.65% of outputs generated by industry leading models contained some form of bias, highlighting a substantial risk of using these models in critical decision making systems. BEATS framework and benchmark offer a scalable and statistically rigorous methodology to benchmark LLMs, diagnose factors driving biases, and develop mitigation strategies. With the BEATS framework, our goal is to help the development of more socially responsible and ethically aligned AI models.