# A Matter of Framing: The Impact of Linguistic Formalism on Probing Results

arXiv: https://arxiv.org/abs/2004.14999 · HTML: https://mlsanity.com/paper/2004.14999/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 12 of 12)

- [A Non-Linear Structural Probe](https://mlsanity.com/paper/2105.10185/) (2021)
- [How is BERT surprised? Layerwise detection of linguistic anomalies](https://mlsanity.com/paper/2105.07452/) (2021)
- [LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond](https://mlsanity.com/paper/2105.12449/) (2021)
- [Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representations](https://mlsanity.com/paper/2109.05958/) (2021)
- [Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling](https://mlsanity.com/paper/2112.07874/) (2021)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [The Architectural Bottleneck Principle](https://mlsanity.com/paper/2211.06420/) (2022)
- [Sentence Ambiguity, Grammaticality and Complexity Probes](https://mlsanity.com/paper/2210.06928/) (2022)
- [Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings](https://mlsanity.com/paper/2301.08998/) (2023)
- [Semantic Composition in Visually Grounded Language Models](https://mlsanity.com/paper/2305.16328/) (2023)
- [Learning Semantic Role Labeling from Compatible Label Sequences](https://mlsanity.com/paper/2305.14600/) (2023)
- [Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities](https://mlsanity.com/paper/2307.01870/) (2023)

## Abstract (arXiv)

Deep pre-trained contextualized encoders like BERT (Delvin et al., 2019) demonstrate remarkable performance on a range of downstream tasks. A recent line of research in probing investigates the linguistic knowledge implicitly learned by these models during pre-training. While most work in probing operates on the task level, linguistic tasks are rarely uniform and can be represented in a variety of formalisms. Any linguistics-based probing study thereby inevitably commits to the formalism used to annotate the underlying data. Can the choice of formalism affect probing results? To investigate, we conduct an in-depth cross-formalism layer probing study in role semantics. We find linguistically meaningful differences in the encoding of semantic role- and proto-role information by BERT depending on the formalism and demonstrate that layer probing can detect subtle differences between the implementations of the same linguistic formalism. Our results suggest that linguistic formalism is an important dimension in probing studies, along with the commonly used cross-task and cross-lingual experimental settings.