mlsanityData to 7 Oct 2026

B-score: Detecting biases in large language models using response history

arXiv 2505.18545 ↗First posted 24 May 2025cs.LG cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) often exhibit strong biases, e.g, against women or in favor of the number 7. We investigate whether LLMs would be able to output less biased answers when allowed to observe their prior answers to the same question in a multi-turn conversation. To understand which types of questions invite more biased answers, we test LLMs on our proposed set of questions that span 9 topics and belong to three types: (1) Subjective; (2) Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves in a multi-turn conversation in response to questions that seek an Random, unbiased answer. Furthermore, we propose B-score, a novel metric that is effective in detecting biases to Subjective, Random, Easy, and Hard questions. On MMLU, HLE, and CSQA, leveraging B-score substantially improves the verification accuracy of LLM answers (i.e, accepting LLM correct answers and rejecting incorrect ones) compared to using verbalized confidence scores or the frequency of single-turn answers alone. Code and data are available at: https://b-score.github.io.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

32 of 36 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  9. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  10. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  11. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  12. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
20 more
  1. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  2. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  3. 2024Cognitive Bias in Decision-Making with LLMscited by 104
  4. 2024Large Language Models are Geographically Biasedcited by 91
  5. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  6. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  7. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 78
  8. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  9. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  10. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  11. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  12. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53
  13. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  14. 2023LitCab: Lightweight Language Model Calibration over Short- and Long-form Responsescited by 44
  15. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  16. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  17. 2024Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgmentscited by 23
  18. 2023GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Modelscited by 16
  19. 2024Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queriescited by 15
  20. 2024Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  2. 2026Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variationscited by 4
  3. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  4. 2026D-Models and E-Models: Diversity-Stability Trade-offs in the Sampling Behavior of Large Language Modelscited by 1
  5. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  6. 2026Deep and shallow biases in language modelscited by 0