mlsanityData to 7 Oct 2026

A Non-Linear Structural Probe

arXiv 2105.10185 ↗First posted 21 May 2021cs.CL cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Probes are models devised to investigate the encoding of knowledge -- e.g. syntactic structure -- in contextual representations. Probes are often designed for simplicity, which has led to restrictions on probe design that may not allow for the full exploitation of the structure of encoded information; one such restriction is linearity. We examine the case of a structural probe (Hewitt and Manning, 2019), which aims to investigate the encoding of syntactic structure in contextual representations through learning only linear transformations. By observing that the structural probe learns a metric, we are able to kernelize it and develop a novel non-linear variant with an identical number of parameters. We test on 6 languages and find that the radial-basis function (RBF) kernel, in conjunction with regularization, achieves a statistically significant improvement over the baseline in all languages -- implying that at least part of the syntactic knowledge is encoded non-linearly. We conclude by discussing how the RBF kernel resembles BERT's self-attention layers and speculate that this resemblance leads to the RBF-based probe's stronger performance.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 13 of its 30 references that are in the map. First posted on arXiv 21 May 2021.

Builds on

13 of 30 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Designing and Interpreting Probes with Control Taskscited by 608
  8. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  9. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  10. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  11. 2020Pareto Probing: Trading Off Accuracy for Complexitycited by 55
  12. 2020A Tale of a Probe and a Parsercited by 40
1 more
  1. 2020A Matter of Framing: The Impact of Linguistic Formalism on Probing Resultscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 4; 2024: 7; 2025: 7; 2026: 6.

31 papers.

Sort:

By area:

  1. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  2. 2022Kernelized Concept Erasurecited by 47
  3. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  4. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  5. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  6. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  7. 2024Mechanistic?cited by 25
  8. 2024A polar coordinate system represents syntax in large language modelscited by 24
  9. 2022AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language modelscited by 16
  10. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 14
  11. 2023Arithmetic with Language Models: from Memorization to Computationcited by 12
  12. 2024Non-Linear Inference Time Intervention: Improving LLM Truthfulnesscited by 9
19 more
  1. 2024How Reliable are Causal Probing Interventions?cited by 9
  2. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  3. 2022The Architectural Bottleneck Principlecited by 5
  4. 2025Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfercited by 5
  5. 2024A Critical Study of What Code-LLMs (Do Not) Learncited by 4
  6. 2024Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Modelscited by 4
  7. 2025Probing Syntax in Large Language Models: Successes and Remaining Challengescited by 4
  8. 2022Emergent Linguistic Structures in Neural Networks are Fragilecited by 2
  9. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  10. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  11. 2026Implicit Representations of Grammaticality in Language Modelscited by 1
  12. 2023Disentangling the Linguistic Competence of Privacy-Preserving BERTcited by 0
  13. 2023Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Modelscited by 0
  14. 2024Hitting "Probe"rty with Non-Linearity, and Morecited by 0
  15. 2025Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Modelscited by 0
  16. 2025Freeze, Diffuse, Decode: Task-Aware Adaptation of Transformer Embeddings for Antimicrobial Peptide Designcited by 0
  17. 2026Probing for Reading Timescited by 0
  18. 2026D2-Monitor: Dynamic Safety Monitoring for Diffusion LLMs via Hesitation Signalscited by 0
  19. 2026A Survey on the Linear Representation Hypothesiscited by 0