mlsanityData to 7 Oct 2026

SODAPOP: Open-Ended Discovery of Social Biases in Social Commonsense Reasoning Models

arXiv 2210.07269 ↗First posted 13 Oct 2022cs.CLcited by 22

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

A common limitation of diagnostic tests for detecting social biases in NLP models is that they may only detect stereotypic associations that are pre-specified by the designer of the test. Since enumerating all possible problematic associations is infeasible, it is likely these tests fail to detect biases that are present in a model but not pre-specified by the designer. To address this limitation, we propose SODAPOP (SOcial bias Discovery from Answers about PeOPle) in social commonsense question-answering. Our pipeline generates modified instances from the Social IQa dataset (Sap et al., 2019) by (1) substituting names associated with different demographic groups, and (2) generating many distractor answers from a masked language model. By using a social commonsense model to score the generated distractors, we are able to uncover the model's stereotypic associations between demographic groups and an open set of words. We also test SODAPOP on debiased models and show the limitations of multiple state-of-the-art debiasing algorithms.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 55 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

38 of 55 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  6. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  8. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  9. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  10. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  11. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  12. 2018Gender Bias in Coreference Resolutioncited by 438
26 more
  1. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  2. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  3. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  4. 2019Measuring Bias in Contextualized Word Representationscited by 275
  5. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  6. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  7. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  8. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  9. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  10. 2020Towards Debiasing Sentence Representationscited by 170
  11. 2020Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biasescited by 167
  12. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  13. 2020Towards Controllable Biases in Language Generationcited by 112
  14. 2019It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitutioncited by 102
  15. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  16. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  17. 2020"You are grounded!": Latent Name Artifacts in Pre-trained Language Modelscited by 39
  18. 2021Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Modelscited by 39
  19. 2016Ordinal Common-sense Inferencecited by 35
  20. 2022Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Modelscited by 35
  21. 2020Unsupervised Discovery of Implicit Gender Biascited by 34
  22. 2018What are the biases in my word embedding?cited by 30
  23. 2020Automatic Distractor Generation for Multiple Choice Questions in Standard Testscited by 26
  24. 2020Knowledge-Driven Distractor Generation for Cloze-style Multiple Choice Questionscited by 24
  25. 2021Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluationcited by 9
  26. 2022Race and ethnicity data for first, middle, and last namescited by 2

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 13; 2025: 4.

22 papers.

Sort:

By area:

  1. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  2. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  3. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  4. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  5. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 39
  6. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  7. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  8. 2024Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?cited by 24
  9. 2024Stop! In the Name of Flaws: Disentangling Personal Names and Sociodemographic Attributes in NLPcited by 22
  10. 2024Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Promptingcited by 22
  11. 2023CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Biascited by 13
  12. 2023Nichelle and Nancy: The Influence of Demographic Attributes and Tokenization Length on First Name Biasescited by 12
10 more
  1. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  2. 2024Fairness Definitions in Language Models Explainedcited by 10
  3. 2024CLIMB: A Benchmark of Clinical Bias in Large Language Modelscited by 7
  4. 2025On the Mutual Influence of Gender and Occupation in LLM Representationscited by 4
  5. 2024With a Grain of SALT: Are LLMs Fair Across Social Dimensions?cited by 3
  6. 2024When Stereotypes GTG: The Impact of Predictive Text Suggestions on Gender Bias in Human-AI Co-Writingcited by 2
  7. 2025Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Modelscited by 2
  8. 2025Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotatorscited by 1
  9. 2024On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Modelscited by 0
  10. 2025Developing A Framework to Support Human Evaluation of Bias in Generated Free Response Textcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating Thinking Behaviours of Reasoning-Based Language Models for Social Bias Mitigation
  2. 2023SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models
  3. 2024BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs
  4. 2024Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings
  5. 2026Evaluating Social Bias in RAG Systems: When External Context Helps and Reasoning Hurts
  6. 2025BiasCause: Evaluate Socially Biased Causal Reasoning of Large Language Models
  7. 2025Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning
  8. 2025An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation