mlsanityData to 7 Oct 2026

On the Evolution of Syntactic Information Encoded by BERT's Contextualized Representations

arXiv 2101.11492 ↗First posted 27 Jan 2021cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

The adaptation of pretrained language models to solve supervised tasks has become a baseline in NLP, and many recent works have focused on studying how linguistic information is encoded in the pretrained sentence representations. Among other information, it has been shown that entire syntax trees are implicitly embedded in the geometry of such models. As these models are often fine-tuned, it becomes increasingly important to understand how the encoded knowledge evolves along the fine-tuning. In this paper, we analyze the evolution of the embedded syntax trees along the fine-tuning process of BERT for six different tasks, covering all levels of the linguistic structure. Experimental results show that the encoded syntactic information is forgotten (PoS tagging), reinforced (dependency and constituency parsing) or preserved (semantics-related tasks) in different ways along the fine-tuning process depending on the task.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 27 Jan 2021.

Builds on

37 of 50 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2018Deep contextualized word representationscited by 2,285
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  7. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  8. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  9. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  10. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  11. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  12. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
25 more
  1. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  2. 2019Revealing the Dark Secrets of BERTcited by 300
  3. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  4. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  5. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  6. 2019Assessing BERT's Syntactic Abilitiescited by 191
  7. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  8. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  9. 2019Simple BERT Models for Relation Extraction and Semantic Role Labelingcited by 154
  10. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  11. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  12. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  13. 2020Syntactic Structure from Deep Learningcited by 99
  14. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  15. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  16. 2020Do Syntax Trees Help Pre-trained Transformers Extract Information?cited by 47
  17. 2019Linking artificial and human neural representations of languagecited by 37
  18. 2020Do Neural Language Models Show Preferences for Syntactic Formalisms?cited by 30
  19. 2021Attention Can Reflect Syntactic Structure (If You Let It)cited by 22
  20. 2020Parsing as Pretrainingcited by 19
  21. 2020Syntactic Structure Distillation Pretraining For Bidirectional Encoderscited by 18
  22. 2020The Unstoppable Rise of Computational Linguistics in Deep Learningcited by 12
  23. 2020Amnesic Probing: Behavioral Explanation with Amnesic Counterfactualscited by 12
  24. 2019How to best use Syntax in Semantic Role Labellingcited by 11
  25. 2020Universal Dependencies according to BERT: both more specific and more generalcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 0; 2024: 1; 2025: 2; 2026: 1.

6 papers.

  1. 2021On the Interplay Between Fine-tuning and Composition in Transformerscited by 6
  2. 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
  3. 2024Interpretability of Language Models via Task Spacescited by 4
  4. 2025A new kid on the block: Distributional semantics predicts the word-specific tone signatures of monosyllabic words in conversational Taiwan Mandarincited by 3
  5. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  6. 2025Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrievalcited by 0