mlsanityData to 7 Oct 2026

Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale

arXiv 2403.08293 ↗First posted 13 Mar 2024cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner. We present Generative Pretrained Structured Transformers (GPST), an unsupervised SLM at scale capable of being pre-trained from scratch on raw texts with high parallelism. GPST circumvents the limitations of previous SLMs such as relying on gold trees and sequential training. It consists of two components, a usual SLM supervised by a uni-directional language modeling loss, and an additional composition model, which induces syntactic parse trees and computes constituent representations, supervised by a bi-directional language modeling loss. We propose a representation surrogate to enable joint parallel training of the two models in a hard-EM fashion. We pre-train GPST on OpenWebText, a corpus with 9 billion tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable size in numerous tasks covering both language understanding and language generation. Meanwhile, GPST also significantly outperforms existing unsupervised SLMs on left-to-right grammar induction, while holding a substantial acceleration on training.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

35 of 61 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
23 more
  1. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  2. 2016Recurrent Neural Network Grammarscited by 109
  3. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 101
  4. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  5. 2017Structured Attention Networkscited by 95
  6. 2019Compound Probabilistic Context-Free Grammars for Grammar Inductioncited by 76
  7. 2014Grammar as a Foreign Languagecited by 70
  8. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  9. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  10. 2019Unsupervised Recurrent Neural Network Grammarscited by 47
  11. 2015Transition-Based Dependency Parsing with Stack Long Short-Term Memorycited by 45
  12. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  13. 2016Learning to Compose Words into Sentences with Reinforcement Learningcited by 30
  14. 2017Effective Inference for Generative Neural Parsingcited by 24
  15. 2017Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMscited by 23
  16. 2019Ordered Memorycited by 20
  17. 2021R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modelingcited by 19
  18. 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 18
  19. 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
  20. 2022Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representationcited by 14
  21. 2023Augmenting Transformers with Recursively Composed Multi-grained Representationscited by 5
  22. 2018Dependency Grammar Induction with a Neural Variational Transition-based Parsercited by 4
  23. 2023A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 3.

9 papers.

  1. 2024Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modelingcited by 8
  2. 2024Sneaking Syntax into Transformer Language Models with Tree Regularizationcited by 3
  3. 2025A Systematic Study of Compositional Syntactic Transformer Language Modelscited by 1
  4. 2024Unsupervised Morphological Tree Tokenizercited by 0
  5. 2024On Eliciting Syntax from Language Models via Hashingcited by 0
  6. 2025Understanding Syntactic Generalization in Structure-inducing Language Modelscited by 0
  7. 2026Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMscited by 0
  8. 2026GiLT: Augmenting Transformer Language Models with Dependency Graphscited by 0
  9. 2026Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammarscited by 0