mlsanityData to 14 Oct 2026

Teaching and Critiquing Conceptualization and Operationalization in NLP

arXiv 2512.18505 ↗First posted 20 Dec 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

NLP researchers regularly invoke abstract concepts like "interpretability," "bias," "reasoning," and "stereotypes," without defining them. Each subfield has a shared understanding or conceptualization of what these terms mean and how we should treat them, and this shared understanding is the basis on which operational decisions are made: Datasets are built to evaluate these concepts, metrics are proposed to quantify them, and claims are made about systems. But what do they mean, what should they mean, and how should we measure them? I outline a seminar I created for students to explore these questions of conceptualization and operationalization, with an interdisciplinary reading list and an emphasis on discussion and critique.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 77 references that are in the map. First posted on arXiv 20 Dec 2025.

Builds on

35 of 77 references.

References · 35

  1. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  2. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  3. 2019Attention is not Explanationcited by 929
  4. 2019Attention is not not Explanationcited by 636
  5. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  6. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  7. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  8. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  9. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  10. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  11. 2021The Values Encoded in Machine Learning Researchcited by 172
  12. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
23 more
  1. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  2. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  4. 2017Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciencescited by 85
  5. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 79
  6. 2021A Survey of Race, Racism, and Anti-Racism in NLPcited by 75
  7. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  8. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  9. 2022How Large Language Models are Transforming Machine-Paraphrased Plagiarismcited by 38
  10. 2021Disembodied Machine Learning: On the Illusion of Objectivity in NLPcited by 37
  11. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  12. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  13. 2023Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Modelscited by 27
  14. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 26
  15. 2025Cognitive Foundations for Reasoning and Their Manifestation in LLMscited by 26
  16. 2024Stop! In the Name of Flaws: Disentangling Personal Names and Sociodemographic Attributes in NLPcited by 22
  17. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  18. 2023Nichelle and Nancy: The Influence of Demographic Attributes and Tokenization Length on First Name Biasescited by 12
  19. 2024WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Casecited by 9
  20. 2024The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classificationcited by 5
  21. 2024Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distributioncited by 4
  22. 2025EuroGEST: Investigating gender stereotypes in multilingual language modelscited by 4
  23. 2024Training an NLP Scholar at a Small Liberal Arts College: A Backwards Designed Course Proposalcited by 1

Built on by

Not cited yet.