mlsanityData to 7 Oct 2026

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling

arXiv 2012.00857 ↗First posted 1 Dec 2020cs.CL cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one or several corresponded words. While previous unsupervised parsing methods mostly focus on only inducing one class of grammars, we introduce a novel model, StructFormer, that can simultaneously induce dependency and constituency structure. To achieve this, we propose a new parsing framework that can jointly generate a constituency tree and dependency graph. Then we integrate the induced dependency relations into the transformer, in a differentiable manner, through a novel dependency-constrained self-attention mechanism. Experimental results show that our model can achieve strong results on unsupervised constituency parsing, unsupervised dependency parsing, and masked language modeling at the same time.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 11; 2024: 6; 2025: 4; 2026: 1.

33 papers.

Sort:

By area:

  1. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  2. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  3. 2021Neural Bi-Lexicalized PCFG Inductioncited by 18
  4. 2021Modeling Hierarchical Structures with Continuous Recursive Neural Networkscited by 15
  5. 2021Semantic Role Labeling as Dependency Parsing: Exploring Latent Tree Structures Inside Argumentscited by 14
  6. 2022Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representationcited by 14
  7. 2023Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attentioncited by 12
  8. 2022Dynamic Programming in Rank Space: Scaling Structured Inference with Low-Rank HMMs and PCFGscited by 10
  9. 2023LVDiffusor: Distilling Functional Rearrangement Priors from Large Models into Diffusorcited by 7
  10. 2024Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Modelscited by 7
  11. 2023Beam Tree Recursive Cellscited by 6
  12. 2021Cooperative Self-training of Machine Reading Comprehensioncited by 5
21 more
  1. 2023Augmenting Transformers with Recursively Composed Multi-grained Representationscited by 5
  2. 2023A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classificationcited by 4
  3. 2023Efficient Beam Tree Recursioncited by 4
  4. 2023Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representationcited by 4
  5. 2024Graph Parsing Networkscited by 4
  6. 2023Simple Hardware-Efficient PCFGs with Independent Left and Right Productionscited by 3
  7. 2023Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Buildingcited by 3
  8. 2024Punctuation Restoration Improves Structure Understanding Without Supervisioncited by 3
  9. 2023The Emergence of Chunking Structures with Hierarchical RNNcited by 2
  10. 2022Dependency-based Mixture Language Modelscited by 1
  11. 2024Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Inductioncited by 1
  12. 2024Improving Unsupervised Constituency Parsing via Maximizing Semantic Informationcited by 1
  13. 2025A Systematic Study of Compositional Syntactic Transformer Language Modelscited by 1
  14. 2021Unsupervised Full Constituency Parsing with Neighboring Distribution Divergencecited by 0
  15. 2021FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encodingcited by 0
  16. 2022Syntactic Inductive Biases for Deep Learning Methodscited by 0
  17. 2022Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?cited by 0
  18. 2024On Eliciting Syntax from Language Models via Hashingcited by 0
  19. 2025Understanding Syntactic Generalization in Structure-inducing Language Modelscited by 0
  20. 2025Probability Distribution Collapse: A Critical Bottleneck to Compact Unsupervised Neural Grammar Inductioncited by 0
  21. 2026Holographic Neural PCFG for Unsupervised Parsingcited by 0