mlsanityData to 7 Oct 2026

A Tale of a Probe and a Parser

arXiv 2005.01641 ↗First posted 4 May 2020cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

Measuring what linguistic information is encoded in neural models of language has become popular in NLP. Researchers approach this enterprise by training "probes" - supervised models designed to extract linguistic structure from another model's output. One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations. The structural probe has a novel design, unattested in the parsing literature, the precise benefit of which is not immediately obvious. To explore whether syntactic probes would do better to make use of existing techniques, we compare the structural probe to a more traditional parser with an identical lightweight parameterisation. The parser outperforms structural probe on UUAS in seven of nine analysed languages, often by a substantial amount (e.g. by 11.1 points in English). Under a second less common metric, however, there is the opposite trend - the structural probe outperforms the parser. This begs the question: which metric should we prefer?

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 12; 2023: 4; 2024: 7; 2025: 3; 2026: 2.

40 papers.

Sort:

By area:

  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  3. 2022Probing for the Usage of Grammatical Numbercited by 57
  4. 2021Probing BERT in Hyperbolic Spacescited by 50
  5. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  6. 2021What if This Modified That? Syntactic Interventions via Counterfactual Embeddingscited by 39
  7. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  8. 2021A Non-Linear Structural Probecited by 31
  9. 2021Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probingcited by 28
  10. 2021A Bayesian Framework for Information-Theoretic Probingcited by 22
  11. 2022AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language modelscited by 16
  12. 2025Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViTcited by 16
28 more
  1. 2022Probing for Constituency Structure in Neural Language Modelscited by 13
  2. 2022Predicting Fine-Tuning Performance with Probingcited by 13
  3. 2022When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probescited by 12
  4. 2022Probing for Labeled Dependency Treescited by 8
  5. 2021The Rediscovery Hypothesis: Language Models Need to Meet Linguisticscited by 7
  6. 2022Evaluating the Construct Validity of Text Embeddings with Application to Survey Questionscited by 7
  7. 2022Latent Topology Induction for Understanding Contextualized Representationscited by 7
  8. 2023How Well Do Large Language Models Understand Syntax? An Evaluation by Asking Natural Language Questionscited by 7
  9. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  10. 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
  11. 2022A Latent-Variable Model for Intrinsic Probingcited by 5
  12. 2022The Architectural Bottleneck Principlecited by 5
  13. 2024Probing the Category of Verbal Aspect in Transformer Language Modelscited by 5
  14. 2022Visualizing and Explaining Language Modelscited by 4
  15. 2024A Critical Study of What Code-LLMs (Do Not) Learncited by 4
  16. 2024Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Modelscited by 4
  17. 2021Automatic Expansion of Domain-Specific Affective Models for Web Intelligence Applicationscited by 2
  18. 2022Sort by Structure: Language Model Ranking as Dependency Probingcited by 2
  19. 2023Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Modelscited by 2
  20. 2021Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probescited by 1
  21. 2024What do Transformers Know about Government?cited by 1
  22. 2024DeepCodeProbe: Towards Understanding What Models Trained on Code Learncited by 1
  23. 2024Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Datacited by 1
  24. 2024A Comprehensive Evaluation of Semantic Relation Knowledge of Pretrained Language Models and Humanscited by 1
  25. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  26. 2026Towards Understanding What State Space Models Learn About Codecited by 1
  27. 2023Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Modelscited by 0
  28. 2025Example-Based Concept Analysis Framework for Deep Weather Forecast Modelscited by 0