mlsanityData to 7 Oct 2026

Beyond Performance: Quantifying and Mitigating Label Bias in LLMs

arXiv 2405.02743 ↗First posted 4 May 2024cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown remarkable adaptability to diverse tasks, by leveraging context prompts containing instructions, or minimal input-output examples. However, recent work revealed they also exhibit label bias -- an undesirable preference toward predicting certain answers over others. Still, detecting and measuring this bias reliably and at scale has remained relatively unexplored. In this study, we evaluate different approaches to quantifying label bias in a model's predictions, conducting a comprehensive investigation across 279 classification tasks and ten LLMs. Our investigation reveals substantial label bias in models both before and after debiasing attempts, as well as highlights the importance of outcomes-based evaluation metrics, which were not previously used in this regard. We further propose a novel label bias calibration method tailored for few-shot prompting, which outperforms recent calibration approaches for both improving performance and mitigating label bias. Our results emphasize that label bias in the predictions of LLMs remains a barrier to their reliability.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

31 of 43 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
19 more
  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  2. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  5. 2023Larger language models do in-context learning differentlycited by 386
  6. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  7. 2022Data Contamination: From Memorization to Exploitationcited by 234
  8. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  9. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  10. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  11. 2023Mitigating Label Biases for In-context Learningcited by 88
  12. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  13. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  14. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  15. 2022Prototypical Verbalizer for Prompt-based Few-shot Tuningcited by 53
  16. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  17. 2020Robustness May Be at Odds with Fairness: An Empirical Study on Class-wise Accuracycited by 45
  18. 2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33
  19. 2023Generative Calibration for In-context Learningcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 7.

24 papers.

Sort:

By area:

  1. 2026Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agentscited by 22
  2. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
  3. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  4. 2024Self-Recognition in Language Modelscited by 17
  5. 2024Mitigating Selection Bias with Node Pruning and Auxiliary Optionscited by 12
  6. 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
  7. 2025Hearing the Order: Investigating Position Bias in Large Audio-Language Modelscited by 4
  8. 2025Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Modelscited by 4
  9. 2025Towards AI-assisted Academic Writingcited by 3
  10. 2025PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generationcited by 3
  11. 2026Revisiting Prompt Sensitivity in Large Language Models for Text Classification: The Role of Prompt Underspecificationcited by 3
  12. 2026Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPOcited by 3
12 more
  1. 2025Leveraging In-Context Learning for Political Bias Testing of LLMscited by 2
  2. 2025Aligning Black-box Language Models with Human Judgmentscited by 1
  3. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  4. 2025Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approachcited by 1
  5. 2025Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choicescited by 1
  6. 2026RBCorr: Response Bias Correction in Language Modelscited by 1
  7. 2026Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generationcited by 1
  8. 2026Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?cited by 1
  9. 2025Exploring Imbalanced Annotations for Effective In-Context Learningcited by 0
  10. 2025Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Marketscited by 0
  11. 2025Metric assessment protocol in the context of answer fluctuation on MCQ taskscited by 0
  12. 2026When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Modelscited by 0