mlsanityData to 7 Oct 2026

Dependency-based Mixture Language Models

arXiv 2203.10256 ↗First posted 19 Mar 2022cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Various models have been proposed to incorporate knowledge of syntactic structures into neural language models. However, previous works have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2. In this paper, we introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention. Extensive experiments and human evaluations show that our method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 19 Mar 2022.

Builds on

39 of 58 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  11. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  12. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
27 more
  1. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
  2. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  3. 2016Recurrent Neural Network Grammarscited by 109
  4. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  5. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  6. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  7. 2018Language GANs Falling Shortcited by 97
  8. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
  9. 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
  10. 2020UNION: An Unreferenced Metric for Evaluating Open-ended Story Generationcited by 60
  11. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  12. 2019Unsupervised Recurrent Neural Network Grammarscited by 47
  13. 2019Head-Driven Phrase Structure Grammar Parsing on Penn Treebankcited by 43
  14. 2016Recurrent Highway Networkscited by 36
  15. 2020StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modelingcited by 33
  16. 2020Syntax-Enhanced Pre-trained Modelcited by 29
  17. 2016Tree-to-Sequence Attentional Neural Machine Translationcited by 28
  18. 2021Stylized Story Generation with Style-Guided Planningcited by 21
  19. 2018On Accurate Evaluation of GANs for Language Generationcited by 20
  20. 2019Syntax-Infused Variational Autoencoder for Text Generationcited by 19
  21. 2017Fraternal Dropoutcited by 14
  22. 2019PaLM: A Hybrid Parser and Language Modelcited by 13
  23. 2018Direct Output Connection for a High-Rank Language Modelcited by 10
  24. 2015Dependency Recurrent Neural Language Models for Sentence Completioncited by 9
  25. 2020Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approachcited by 8
  26. 2020Syntax-driven Iterative Expansion Language Models for Controllable Text Generationcited by 6
  27. 2020Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oraclecited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6