mlsanityData to 7 Oct 2026

From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions

arXiv 1906.01958 ↗First posted 5 Jun 2019cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

We inspect the multi-head self-attention in Transformer NMT encoders for three source languages, looking for patterns that could have a syntactic interpretation. In many of the attention heads, we frequently find sequences of consecutive states attending to the same position, which resemble syntactic phrases. We propose a transparent deterministic method of quantifying the amount of syntactic information present in the self-attentions, based on automatically building and evaluating phrase-structure trees from the phrase-like sequences. We compare the resulting trees to existing constituency treebanks, both manually and by computing precision and recall.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 5; 2024: 2; 2025: 2; 2026: 2.

22 papers.

Sort:

By area:

  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  3. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignmentcited by 17
  4. 2021Predicting Discourse Trees from Transformer-based Neural Summarizerscited by 10
  5. 2021Linguistic Dependencies and Statistical Dependencecited by 10
  6. 2022What does Transformer learn about source code?cited by 9
  7. 2023Transformer Language Models Handle Word Frequency in Prediction Headcited by 8
  8. 2021GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networkscited by 7
  9. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  10. 2022Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Modelscited by 6
  11. 2026Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernelcited by 5
  12. 2023Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialoguescited by 4
10 more
  1. 2023CUE: An Uncertainty Interpretation Framework for Text Classifiers Built on Pre-Trained Language Modelscited by 3
  2. 2024Embedded Named Entity Recognition using Probing Classifierscited by 3
  3. 2022Emergent Linguistic Structures in Neural Networks are Fragilecited by 2
  4. 2022Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Modelscited by 1
  5. 2023Semantic Composition in Visually Grounded Language Modelscited by 1
  6. 2023Where exactly does contextualization in a PLM happen?cited by 1
  7. 2025Multilingual Vision-Language Models, A Surveycited by 1
  8. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  9. 2021Focused Contrastive Training for Test-based Constituency Analysiscited by 0
  10. 2024On Eliciting Syntax from Language Models via Hashingcited by 0