mlsanityData to 14 Oct 2026

Intrinsic Bias Metrics Do Not Correlate with Application Bias

arXiv 2012.15859 ↗First posted 31 Dec 2020cs.CLcited by 148

Not on the map. Why?

Abstract (arXiv)

Natural Language Processing (NLP) systems learn harmful societal biases that cause them to amplify inequality as they are deployed in more and more situations. To guide efforts at debiasing these systems, the NLP community relies on a variety of metrics that quantify bias in models. Some of these metrics are intrinsic, measuring bias in word embedding spaces, and some are extrinsic, measuring bias in downstream tasks that the word embeddings enable. Do these intrinsic and extrinsic metrics correlate with each other? We compare intrinsic and extrinsic metrics across hundreds of trained models covering different tasks and experimental conditions. Our results show no reliable correlation between these metrics that holds in all scenarios across tasks and languages. We urge researchers working on debiasing to focus on extrinsic measures of bias, and to make using these measures more feasible via creation of new challenge sets and annotated test data. To aid this effort, we release code, a new intrinsic metric, and an annotated test set focused on gender bias in hate speech.

Key facts. Cited by 148 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 32; 2023: 30; 2024: 34; 2025: 27; 2026: 17.

148 papers.

Sort:

By area:

  1. 2023PaLM 2 Technical Reportcited by 1,162
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  3. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  4. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  5. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 178
  6. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  7. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  8. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 149
  9. 2021Gender Bias in Machine Translationcited by 117
  10. 2022Linear Adversarial Concept Erasurecited by 101
  11. 2023A Survey on Fairness in Large Language Modelscited by 95
  12. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
38 more
  1. 2021On Measures of Biases and Harms in NLPcited by 81
  2. 2022On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representationscited by 75
  3. 2022Theories of "Gender" in NLP Bias Researchcited by 62
  4. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 58
  5. 2022Perturbation Augmentation for Fairer NLPcited by 52
  6. 2024Measuring Implicit Bias in Explicitly Unbiased Large Language Modelscited by 51
  7. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  8. 2022Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Taskscited by 48
  9. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  10. 2024Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluationcited by 41
  11. 2022Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversalcited by 40
  12. 2024Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attributioncited by 40
  13. 2021Your fairness may vary: Pretrained language model fairness in toxic text classificationcited by 39
  14. 2023Overwriting Pretrained Bias with Finetuning Datacited by 32
  15. 2025Linear Representations of Political Perspective Emerge in Large Language Modelscited by 32
  16. 2022The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarkscited by 31
  17. 2022MABEL: Attenuating Gender Bias using Textual Entailment Datacited by 31
  18. 2022Choose Your Lenses: Flaws in Gender Bias Evaluationcited by 29
  19. 2023On the Independence of Association Bias and Empirical Fairness in Language Modelscited by 29
  20. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 28
  21. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  22. 2024Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gapscited by 25
  23. 2024Investigating Implicit Bias in Large Language Models: A Large-Scale Study of Over 50 LLMscited by 25
  24. 2022A Robust Bias Mitigation Procedure Based on the Stereotype Content Modelcited by 24
  25. 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
  26. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  27. 2024The Impossibility of Fair LLMscited by 22
  28. 2022ADEPT: A DEbiasing PrompT Frameworkcited by 21
  29. 2021Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Modelscited by 20
  30. 2022Challenges in Applying Explainability Methods to Improve the Fairness of NLP Modelscited by 20
  31. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  32. 2021Alexa, Google, Siri: What are Your Pronouns? Gender and Anthropomorphism in the Design and Perception of Conversational Assistantscited by 16
  33. 2022The Birth of Bias: A case study on the evolution of gender bias in an English language modelcited by 16
  34. 2023Fairness in Language Models Beyond English: Gaps and Challengescited by 16
  35. 2023Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labelscited by 16
  36. 2023Co2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuningcited by 15
  37. 2024Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistencycited by 15
  38. 2024Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Biascited by 15

50 most cited of 148 All 148 (JSON)