mlsanityData to 14 Oct 2026

Operationalising Representation in Natural Language Processing

arXiv 2306.08193 ↗First posted 14 Jun 2023cs.CL cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Despite its centrality in the philosophy of cognitive science, there has been little prior philosophical work engaging with the notion of representation in contemporary NLP practice. This paper attempts to fill that lacuna: drawing on ideas from cognitive science, I introduce a framework for evaluating the representational claims made about components of neural NLP models, proposing three criteria with which to evaluate whether a component of a model represents a property and operationalising these criteria using probing classifiers, a popular analysis technique in NLP (and deep learning more broadly). The project of operationalising a philosophically-informed notion of representation should be of interest to both philosophers of science and NLP practitioners. It affords philosophers a novel testing-ground for claims about the nature of representation, and helps NLPers organise the large literature on probing experiments, suggesting novel avenues for empirical research.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 45 of its 90 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

45 of 90 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  11. 2018Neural Network Acceptability Judgmentscited by 958
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
33 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  2. 2016The Mythos of Model Interpretabilitycited by 721
  3. 2019Designing and Interpreting Probes with Control Taskscited by 640
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  5. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  6. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  7. 2018Gender Bias in Coreference Resolutioncited by 442
  8. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  9. 2021Causal Abstractions of Neural Networkscited by 422
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  11. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  12. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  13. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  14. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  15. 2020A Theory of Usable Information Under Computational Constraintscited by 162
  16. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  17. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  18. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  19. 2022Meaning without reference in large language modelscited by 76
  20. 2021Conditional probing: measuring usable information beyond a baselinecited by 62
  21. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 59
  22. 2022Probing for the Usage of Grammatical Numbercited by 59
  23. 2018Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Taskscited by 56
  24. 2020Pareto Probing: Trading Off Accuracy for Complexitycited by 55
  25. 2021Neural population geometry: An approach for understanding biological and artificial neural networkscited by 52
  26. 2021What if This Modified That? Syntactic Interventions via Counterfactual Embeddingscited by 39
  27. 2022Unit Testing for Concepts in Neural Networkscited by 23
  28. 2019Posing Fair Generalization Tasks for Natural Language Inferencecited by 22
  29. 2021A Bayesian Framework for Information-Theoretic Probingcited by 22
  30. 2020An information theoretic view on selecting linguistic probescited by 19
  31. 2020Amnesic Probing: Behavioral Explanation with Amnesic Counterfactualscited by 14
  32. 2022When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probescited by 12
  33. 2020Probing Neural Dialog Models for Conversational Understandingcited by 10

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 7.

14 papers.

Sort:

By area:

  1. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 78
  2. 2024Standards for Belief Representations in LLMscited by 30
  3. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
  4. 2024Does ChatGPT Have a Mind?cited by 14
  5. 2025How Causal Abstraction Underpins Computational Explanationcited by 11
  6. 2024What is it for a Machine Learning Model to Have a Capability?cited by 10
  7. 2025A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.icited by 6
  8. 2025Mechanistic Interpretability Needs Philosophycited by 6
  9. 2025Can structural correspondences ground real world representational content in Large Language Models?cited by 3
  10. 2024Report on Candidate Computational Indicators for Conscious Valenced Experiencecited by 2
  11. 2024Chatting with Bots: AI, Speech Acts, and the Edge of Assertioncited by 2
  12. 2025What Is AI Safety? What Do We Want It to Be?cited by 2
2 more
  1. 2025Representation in large language modelscited by 0
  2. 2025Explaining Neural Networks with Reasonscited by 0