mlsanityData to 7 Oct 2026

Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees

arXiv 2103.04350 ↗First posted 7 Mar 2021cs.CLcited by 51

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models like BERT achieve superior performances in various NLP tasks without explicit consideration of syntactic information. Meanwhile, syntactic information has been proved to be crucial for the success of NLP applications. However, how to incorporate the syntax trees effectively and efficiently into pre-trained Transformers is still unsettled. In this paper, we address this problem by proposing a novel framework named Syntax-BERT. This framework works in a plug-and-play mode and is applicable to an arbitrary pre-trained checkpoint based on Transformer architecture. Experiments on various datasets of natural language understanding verify the effectiveness of syntax trees and achieve consistent improvement over multiple pre-trained models, including BERT, RoBERTa, and T5.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 7 Mar 2021.

Builds on

19 of 31 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  10. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  11. 2019Star-Transformercited by 94
  12. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
7 more
  1. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  2. 2020Do Syntax Trees Help Pre-trained Transformers Extract Information?cited by 47
  3. 2015Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Pathcited by 35
  4. 2020Tree-structured Attention with Hierarchical Accumulationcited by 30
  5. 2019Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understandingcited by 20
  6. 2018Scheduled Multi-Task Learning: From Syntax to Translationcited by 18
  7. 2019DeGNN: Characterizing and Improving Graph Neural Networks with Graph Decompositioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 14; 2023: 13; 2024: 8; 2025: 2; 2026: 5.

51 papers.

Sort:

By area:

  1. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  2. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  3. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  4. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  5. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  6. 2022A Survey of Knowledge-Intensive NLP with Pre-Trained Language Modelscited by 30
  7. 2023Summarizing Multiple Documents with Conversational Structure for Meta-Review Generationcited by 27
  8. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24
  9. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  10. 2022DABERT: Dual Attention Enhanced BERT for Semantic Matchingcited by 21
  11. 2023Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflictscited by 21
  12. 2022Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusioncited by 18
38 more
  1. 2023CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Taskscited by 18
  2. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignmentcited by 17
  3. 2022InforMask: Unsupervised Informative Masking for Language Model Pretrainingcited by 15
  4. 2022Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledgecited by 11
  5. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  6. 2021Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modelingcited by 10
  7. 2024Comateformer: Combined Attention Transformer for Semantic Sentence Matchingcited by 9
  8. 2021Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentationcited by 6
  9. 2021Dependency Parsing with Bottom-up Hierarchical Pointer Networkscited by 5
  10. 2021Improving BERT Pretraining with Syntactic Supervisioncited by 4
  11. 2021Discontinuous Grammar as a Foreign Languagecited by 4
  12. 2022Not Just Plain Text! Fuel Document-Level Relation Extraction with Explicit Syntax Refinement and Subsentence Modelingcited by 4
  13. 2024Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervisioncited by 4
  14. 2021Should Semantic Vector Composition be Explicit? Can it be Linear?cited by 3
  15. 2021A Survey of Knowledge Enhanced Pre-trained Modelscited by 3
  16. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  17. 2023CSED: A Chinese Semantic Error Diagnosis Corpuscited by 3
  18. 2023Fusing Temporal Graphs into Transformers for Time-Sensitive Question Answeringcited by 3
  19. 2023Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Modelscited by 2
  20. 2023DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integrationcited by 2
  21. 2024Infusing Prompts with Syntax and Semanticscited by 2
  22. 2022Lite Unified Modeling for Discriminative Reading Comprehensioncited by 1
  23. 2023Semantic Composition in Visually Grounded Language Modelscited by 1
  24. 2023Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-trainingcited by 1
  25. 2024ConcEPT: Concept-Enhanced Pre-Training for Language Modelscited by 1
  26. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  27. 2022Improving Pre-trained Language Models with Syntactic Dependency Prediction Task for Chinese Semantic Error Recognitioncited by 0
  28. 2022Syntax-driven Data Augmentation for Named Entity Recognitioncited by 0
  29. 2023Combining EEG and NLP Features for Predicting Students' Lecture Comprehension using Ensemble Classificationcited by 0
  30. 2024Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It)cited by 0
  31. 2024Negation Triplet Extraction with Syntactic Dependency and Semantic Consistencycited by 0
  32. 2024On Eliciting Syntax from Language Models via Hashingcited by 0
  33. 2024Leveraging Grammar Induction for Language Understanding and Generationcited by 0
  34. 2025Using External knowledge to Enhanced PLM for Semantic Matchingcited by 0
  35. 2025Boosting Neural Language Inference via Cascaded Interactive Reasoningcited by 0
  36. 2026Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMscited by 0
  37. 2026LASQ: A Low-resource Aspect-based Sentiment Quadruple Extraction Datasetcited by 0
  38. 2026Exploiting Pre-trained Encoder-Decoder Transformers for Sequence-to-Sequence Constituent Parsingcited by 0

50 most cited of 51 All 51 (JSON)