mlsanityData to 14 Oct 2026

Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling

arXiv 2112.07874 ↗First posted 15 Dec 2021cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

We examine the extent to which, in principle, linguistic graph representations can complement and improve neural language modeling. With an ensemble setup consisting of a pretrained Transformer and ground-truth graphs from one of 7 different formalisms, we find that, overall, semantic constituency structures are most useful to language modeling performance -- outpacing syntactic constituency structures as well as syntactic and semantic dependency structures. Further, effects vary greatly depending on part-of-speech class. In sum, our findings point to promising tendencies in neuro-symbolic language modeling and invite future research quantifying the design choices made by different formalisms.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 35 of its 66 references that are in the map. First posted on arXiv 15 Dec 2021.

Builds on

35 of 66 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  7. 2019Designing and Interpreting Probes with Control Taskscited by 640
  8. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  9. 2021Causal Abstractions of Neural Networkscited by 422
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  11. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  12. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 306
23 more
  1. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  2. 2016Recurrent Neural Network Grammarscited by 109
  3. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  4. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  5. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  6. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  7. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  8. 2021Syntax-BERT: Improving Pre-trained Transformers with Syntax Treescited by 51
  9. 2019Probing What Different NLP Tasks Teach Machines about Function Word Comprehensioncited by 44
  10. 2020Do Neural Language Models Show Preferences for Syntactic Formalisms?cited by 30
  11. 2018Syntactic Scaffolds for Semantic Structurescited by 27
  12. 2021Structural Guidance for Transformer Language Modelscited by 25
  13. 2020Strongly Incremental Constituency Parsing with Graph Neural Networkscited by 23
  14. 2020Infusing Finetuning with Semantic Dependenciescited by 18
  15. 2020(Re)construing Meaning in NLPcited by 16
  16. 2016Neural Semantic Role Labeling with Dependency Path Embeddingscited by 15
  17. 2019PaLM: A Hybrid Parser and Language Modelcited by 13
  18. 2020A Matter of Framing: The Impact of Linguistic Formalism on Probing Resultscited by 12
  19. 2015Dependency Recurrent Neural Language Models for Sentence Completioncited by 9
  20. 2021Semantics-aware Attention Improves Neural Machine Translationcited by 5
  21. 2019Semantically Constrained Multilayer Annotation: The Case of Coreferencecited by 3
  22. 2019Made for Each Other: Broad-coverage Semantic Structures Meet Preposition Supersensescited by 2
  23. 2020Comparison by Conversion: Reverse-Engineering UCCA from Syntax and Lexical Semanticscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 1; 2026: 3.

10 papers.

  1. 2024Natural Language Processing RELIES on Linguisticscited by 9
  2. 2024Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Modelscited by 8
  3. 2023Linguistic representations for fewer-shot relation extraction across domainscited by 4
  4. 2022An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Taskscited by 2
  5. 2023Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Modelscited by 2
  6. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  7. 2026Speaking of Language: Reflections on Metalanguage Research in NLPcited by 1
  8. 2026GiLT: Augmenting Transformer Language Models with Dependency Graphscited by 1
  9. 2023Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structurescited by 0
  10. 2025Exploring Graph Representations of Logical Forms for Language Modelingcited by 0