mlsanityData to 7 Oct 2026

Designing and Interpreting Probes with Control Tasks

arXiv 1909.03368 ↗First posted 8 Sep 2019cs.CLcited by 608

Not on the map. Why?

Cited by 32 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Probes, supervised models trained to predict properties (like parts-of-speech) from representations (like ELMo), have achieved high accuracy on a range of linguistic tasks. But does this mean that the representations encode linguistic structure or just that the probe has learned the linguistic task? In this paper, we propose control tasks, which associate word types with random outputs, to complement linguistic tasks. By construction, these tasks can only be learned by the probe itself. So a good probe, (one that reflects the representation), should be selective, achieving high linguistic task accuracy and low control task accuracy. The selectivity of a probe puts linguistic task accuracy in context with the probe's capacity to memorize from word types. We construct control tasks for English part-of-speech tagging and dependency edge prediction, and show that popular probes on ELMo representations are not selective. We also find that dropout, commonly used to control probe complexity, is ineffective for improving selectivity of MLPs, but that other forms of regularization are effective. Finally, we find that while probes on the first layer of ELMo yield slightly better part-of-speech tagging accuracy than the second, probes on the second layer are substantially more selective, which raises the question of which layer better represents parts-of-speech.

Key facts. Cited by 608 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 90; 2022: 68; 2023: 70; 2024: 64; 2025: 68; 2026: 248.

608 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  3. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  5. 2021Causal Abstractions of Neural Networkscited by 401
  6. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  7. 2023Explainability for Large Language Models: A Surveycited by 331
  8. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  9. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  10. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  11. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 225
  12. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
38 more
  1. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  2. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  3. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  4. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  5. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  6. 2022A Review on Language Models as Knowledge Basescited by 156
  7. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  8. 2024How do Large Language Models Handle Multilingualism?cited by 148
  9. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  10. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  11. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  12. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  13. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  14. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  15. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  16. 2024Evaluating the World Model Implicit in a Generative Modelcited by 107
  17. 2023How do Language Models Bind Entities in Context?cited by 102
  18. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  19. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  20. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  21. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  22. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  23. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  24. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  25. 2021Language Model Evaluation Beyond Perplexitycited by 77
  26. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  27. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  28. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  29. 2021Chess as a Testbed for Language Model State Trackingcited by 69
  30. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  31. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  32. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
  33. 2021Low-Complexity Probing via Finding Subnetworkscited by 63
  34. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  35. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  36. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 61
  37. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  38. 2022Probing for the Usage of Grammatical Numbercited by 57

50 most cited of 608 All 608 (JSON)