mlsanityData to 14 Oct 2026

Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP

arXiv 2302.05711 ↗First posted 11 Feb 2023cs.CL cs.CY cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct. However current progress is hampered by a plurality of definitions of bias, means of quantification, and oftentimes vague relation between debiasing algorithms and theoretical measures of bias. This paper seeks to clarify the current situation and plot a course for meaningful progress in fair learning, with two key contributions: (1) making clear inter-relations among the current gamut of methods, and their relation to fairness theory; and (2) addressing the practical problem of model selection, which involves a trade-off between fairness and accuracy and has led to systemic issues in fairness research. Putting them together, we make several recommendations to help shape future work.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 11 Feb 2023.

Builds on

29 of 43 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  5. 2014Certifying and removing disparate impactcited by 715
  6. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 672
  7. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  8. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  9. 2017Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairnesscited by 384
  10. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 364
  11. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  12. 2017Fairness in Criminal Justice Risk Assessments: The State of the Artcited by 315
17 more
  1. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  2. 2018Adversarial Removal of Demographic Attributes from Text Datacited by 170
  3. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  4. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  5. 2020FairBatch: Batch Selection for Model Fairnesscited by 99
  6. 2018An Empirical Study of Rich Subgroup Fairness for Machine Learningcited by 81
  7. 2019Conditional Learning of Fair Representationscited by 78
  8. 2018Towards Robust and Privacy-preserving Text Representationscited by 75
  9. 2019Attenuating Bias in Word Vectorscited by 64
  10. 2019What's in a Name? Reducing Bias in Bios without Access to Protected Attributescited by 51
  11. 2021Diverse Adversaries for Mitigating Bias in Trainingcited by 48
  12. 2020Stereotypical Bias Removal for Hate Speech Detection Task using Knowledge-based Generalizationscited by 40
  13. 2021Balancing out Bias: Achieving Fairness Through Balanced Trainingcited by 34
  14. 2021Evaluating Debiasing Techniques for Intersectional Biasescited by 30
  15. 2022Optimising Equal Opportunity Fairness in Model Trainingcited by 22
  16. 2022De-biasing "bias" measurementcited by 20
  17. 2022Towards Equal Opportunity Fairness through Adversarial Learningcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3; 2025: 5.

11 papers.

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  2. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  3. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 31
  4. 2024CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Modelscited by 15
  5. 2023How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Rankingcited by 8
  6. 2025Fair Text Classification via Transferable Representationscited by 3
  7. 2024Fair Summarization: Bridging Quality and Diversity in Extractive Summariescited by 0
  8. 2025It's complicated. The relationship of algorithmic fairness and non-discrimination provisions for high-risk systems in the EU AI Actcited by 0
  9. 2025Active Learning Methods for Efficient Data Utilization and Model Performance Enhancementcited by 0
  10. 2025Fair-GPTQ: Bias-Aware Quantization for Large Language Modelscited by 0
  11. 2025On the Interplay between Human Label Variation and Model Fairnesscited by 0