mlsanityData to 14 Oct 2026

Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

arXiv 2002.00737 ↗First posted 30 Jan 2020cs.CL cs.AI cs.LGcited by 47

Not on the map. Why?

Abstract (arXiv)

With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 14; 2023: 12; 2024: 4; 2025: 1; 2026: 1.

47 papers.

Sort:

  1. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  2. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  3. 2021Probing BERT in Hyperbolic Spacescited by 50
  4. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  5. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  6. 2024Language Models as Models of Languagecited by 24
  7. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  8. 2022Discovering Salient Neurons in Deep NLP Modelscited by 21
  9. 2021MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extractioncited by 20
  10. 2022OA-Mine: Open-World Attribute Mining for E-Commerce Products with Weak Supervisioncited by 20
  11. 2021Neural Bi-Lexicalized PCFG Inductioncited by 18
  12. 2021Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Modelscited by 14
35 more
  1. 2021UCPhrase: Unsupervised Context-aware Quality Phrase Taggingcited by 14
  2. 2022Unsupervised Slot Schema Induction for Task-oriented Dialogcited by 11
  3. 2021Linguistic Dependencies and Statistical Dependencecited by 10
  4. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  5. 2023Probing LLMs for Joint Encoding of Linguistic Categoriescited by 9
  6. 2024Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Modelscited by 8
  7. 2022Latent Topology Induction for Understanding Contextualized Representationscited by 7
  8. 2023Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parserscited by 7
  9. 2022Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Modelscited by 6
  10. 2022CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structurecited by 6
  11. 2021Have Attention Heads in BERT Learned Constituency Grammar?cited by 5
  12. 2021Improving BERT Pretraining with Syntactic Supervisioncited by 4
  13. 2021Rule Augmented Unsupervised Constituency Parsingcited by 4
  14. 2022An Interpretability Evaluation Benchmark for Pre-trained Language Modelscited by 4
  15. 2022Syntactic Substitutability as Unsupervised Dependency Syntaxcited by 4
  16. 2023Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialoguescited by 4
  17. 2023Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learningcited by 4
  18. 2021Co-training an Unsupervised Constituency Parser with Weak Supervisioncited by 3
  19. 2024Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?cited by 3
  20. 2022Emergent Linguistic Structures in Neural Networks are Fragilecited by 2
  21. 2023StrAE: Autoencoding for Pre-Trained Embeddings using Explicit Structurecited by 2
  22. 2023Construction Grammar and Language Modelscited by 2
  23. 2023The Emergence of Chunking Structures with Hierarchical RNNcited by 2
  24. 2023Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?cited by 2
  25. 2021Analysing the Effect of Masking Length Distribution of MLM: An Evaluation Framework and Case Study on Chinese MRC Datasetscited by 1
  26. 2022Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Taggingcited by 1
  27. 2022Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Modelscited by 1
  28. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  29. 2021Unsupervised Full Constituency Parsing with Neighboring Distribution Divergencecited by 0
  30. 2022Unsupervised and Few-shot Parsing from Pretrained Language Modelscited by 0
  31. 2022Why Do Neural Language Models Still Need Commonsense Knowledge to Handle Semantic Variations in Question Answering?cited by 0
  32. 2023Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?cited by 0
  33. 2023Dynamic Syntax Mapping: A New Approach to Unsupervised Syntax Parsingcited by 0
  34. 2024On Eliciting Syntax from Language Models via Hashingcited by 0
  35. 2025The Dream Within Huang Long Cave: AI-Driven Interactive Narrative for Family Storytelling and Emotional Reflectioncited by 0