# A Non-Linear Structural Probe arXiv: https://arxiv.org/abs/2105.10185 ยท HTML: https://mlsanity.com/paper/2105.10185/ ## Key facts - Cited by 31 papers in the map (data through 14 Oct 2026). - Builds on 13 of its 30 references that are in the map. - First posted on arXiv 21 May 2021. ## Builds on (most cited first, 13 of 13) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016) - [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019) - [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019) - [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020) - [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019) - [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020) - [Pareto Probing: Trading Off Accuracy for Complexity](https://mlsanity.com/paper/2010.02180/) (2020) - [A Tale of a Probe and a Parser](https://mlsanity.com/paper/2005.01641/) (2020) - [A Matter of Framing: The Impact of Linguistic Formalism on Probing Results](https://mlsanity.com/paper/2004.14999/) (2020) ## Built on by (most cited first, 30 of 31) - [Conditional probing: measuring usable information beyond a baseline](https://mlsanity.com/paper/2109.09234/) (2021) - [Kernelized Concept Erasure](https://mlsanity.com/paper/2201.12191/) (2022) - [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021) - [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021) - [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025) - [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023) - [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024) - [A polar coordinate system represents syntax in large language models](https://mlsanity.com/paper/2412.05571/) (2024) - [AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models](https://mlsanity.com/paper/2206.11719/) (2022) - [Beyond Linear Probes: Dynamic Safety Monitoring for Language Models](https://mlsanity.com/paper/2509.26238/) (2025) - [Arithmetic with Language Models: from Memorization to Computation](https://mlsanity.com/paper/2308.01154/) (2023) - [How Reliable are Causal Probing Interventions?](https://mlsanity.com/paper/2408.15510/) (2024) - [Non-Linear Inference Time Intervention: Improving LLM Truthfulness](https://mlsanity.com/paper/2403.18680/) (2024) - [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025) - [The Architectural Bottleneck Principle](https://mlsanity.com/paper/2211.06420/) (2022) - [Probing Syntax in Large Language Models: Successes and Remaining Challenges](https://mlsanity.com/paper/2508.03211/) (2025) - [Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfer](https://mlsanity.com/paper/2511.01023/) (2025) - [A Critical Study of What Code-LLMs (Do Not) Learn](https://mlsanity.com/paper/2406.11930/) (2024) - [Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models](https://mlsanity.com/paper/2412.05353/) (2024) - [Emergent Linguistic Structures in Neural Networks are Fragile](https://mlsanity.com/paper/2210.17406/) (2022) - [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026) - [Perturbation: A simple and efficient adversarial tracer for representation learning in language models](https://mlsanity.com/paper/2603.23821/) (2026) - [Implicit Representations of Grammaticality in Language Models](https://mlsanity.com/paper/2605.05197/) (2026) - [Disentangling the Linguistic Competence of Privacy-Preserving BERT](https://mlsanity.com/paper/2310.11363/) (2023) - [Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Models](https://mlsanity.com/paper/2310.12454/) (2023) - [Hitting "Probe"rty with Non-Linearity, and More](https://mlsanity.com/paper/2402.16168/) (2024) - [Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models](https://mlsanity.com/paper/2506.21861/) (2025) - [Freeze, Diffuse, Decode: Task-Aware Adaptation of Transformer Embeddings for Antimicrobial Peptide Design](https://mlsanity.com/paper/2511.23120/) (2025) - [Probing for Reading Times](https://mlsanity.com/paper/2604.18712/) (2026) - [D2-Monitor: Dynamic Safety Monitoring for Diffusion LLMs via Hesitation Signals](https://mlsanity.com/paper/2605.25893/) (2026) ## Abstract (arXiv) Probes are models devised to investigate the encoding of knowledge -- e.g. syntactic structure -- in contextual representations. Probes are often designed for simplicity, which has led to restrictions on probe design that may not allow for the full exploitation of the structure of encoded information; one such restriction is linearity. We examine the case of a structural probe (Hewitt and Manning, 2019), which aims to investigate the encoding of syntactic structure in contextual representations through learning only linear transformations. By observing that the structural probe learns a metric, we are able to kernelize it and develop a novel non-linear variant with an identical number of parameters. We test on 6 languages and find that the radial-basis function (RBF) kernel, in conjunction with regularization, achieves a statistically significant improvement over the baseline in all languages -- implying that at least part of the syntactic knowledge is encoded non-linearly. We conclude by discussing how the RBF kernel resembles BERT's self-attention layers and speculate that this resemblance leads to the RBF-based probe's stronger performance.