mlsanityData to 7 Oct 2026

Counterfactual Fairness in Text Classification through Robustness

arXiv 1809.10610 ↗First posted 27 Sep 2018cs.LG stat.MLcited by 150

Not on the map. Why?

Abstract (arXiv)

In this paper, we study counterfactual fairness in text classification, which asks the question: How would the prediction change if the sensitive attribute referenced in the example were different? Toxicity classifiers demonstrate a counterfactual fairness issue by predicting that "Some people are gay" is toxic while "Some people are straight" is nontoxic. We offer a metric, counterfactual token fairness (CTF), for measuring this particular form of fairness in text classifiers, and describe its relationship with group fairness. Further, we offer three approaches, blindness, counterfactual augmentation, and counterfactual logit pairing (CLP), for optimizing counterfactual token fairness during training, bridging the robustness and fairness literature. Empirically, we find that blindness and CLP address counterfactual token fairness. The methods do not harm classifier performance, and have varying tradeoffs with group fairness. These approaches, both for measurement and optimization, provide a new path forward for addressing fairness concerns in text classification.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 40; 2023: 35; 2024: 23; 2025: 13; 2026: 10.

150 papers.

Sort:

By area:

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  3. 2022Red Teaming Language Models with Language Modelscited by 983
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  5. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  6. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  7. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  8. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  9. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  10. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  11. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  12. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
38 more
  1. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 102
  2. 2023A Survey on Fairness in Large Language Modelscited by 91
  3. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  4. 2021Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Testscited by 77
  5. 2023Fast Model Debias with Machine Unlearningcited by 70
  6. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  7. 2022Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Modelscited by 51
  8. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  9. 2022Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioningcited by 45
  10. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  11. 2022Causality-based Neural Network Repaircited by 44
  12. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  13. 2021Detecting Cross-Geographic Biases in Toxicity Modeling on Social Mediacited by 31
  14. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimizationcited by 31
  15. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  16. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactualscited by 27
  17. 2022From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible MLcited by 26
  18. 2021Testing Group Fairness via Optimal Transport Projectionscited by 23
  19. 2023Counterfactual Learning on Graphs: A Surveycited by 22
  20. 2022Deep Learning on a Healthy Data Diet: Finding Important Examples for Fairnesscited by 21
  21. 2021Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classificationcited by 20
  22. 2022Learning fair representation with a parametric integral probability metriccited by 20
  23. 2022Challenges in Applying Explainability Methods to Improve the Fairness of NLP Modelscited by 20
  24. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  25. 2022Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detectioncited by 18
  26. 2021Probabilistic Verification of Neural Networks Against Group Fairnesscited by 17
  27. 2021Modeling Techniques for Machine Learning Fairness: A Surveycited by 17
  28. 2023Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Modelscited by 17
  29. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  30. 2022Accurate Fairness: Improving Individual Fairness without Trading Accuracycited by 16
  31. 2021Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventionscited by 15
  32. 2023Safety and Fairness for Content Moderation in Generative Modelscited by 15
  33. 2021Automatic Fairness Testing of Neural Classifiers through Adversarial Samplingcited by 14
  34. 2023Data Augmentations for Improved (Large) Language Model Generalizationcited by 14
  35. 2022Domain Adaptation meets Individual Fairness. And they get alongcited by 13
  36. 2022Flexible text generation for counterfactual fairness probingcited by 13
  37. 2021Individually Fair Gradient Boostingcited by 12
  38. 2023Robustness Implies Fairness in Causal Algorithmic Recoursecited by 12

50 most cited of 150 All 150 (JSON)