# SG-Net: Syntax Guided Transformer for Language Representation

arXiv: https://arxiv.org/abs/2012.13915 · HTML: https://mlsanity.com/paper/2012.13915/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2020.

## Built on by (most cited first, 13 of 13)

- [Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://mlsanity.com/paper/2110.06696/) (2021)
- [Structural Guidance for Transformer Language Models](https://mlsanity.com/paper/2108.00104/) (2021)
- [QSAN: A Near-term Achievable Quantum Self-Attention Network](https://mlsanity.com/paper/2207.07563/) (2022)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [Online Metro Origin-Destination Prediction via Heterogeneous Information Aggregation](https://mlsanity.com/paper/2107.00946/) (2021)
- [Parse Trees Guided LLM Prompt Compression](https://mlsanity.com/paper/2409.15395/) (2024)
- [Discourse-Aware Graph Networks for Textual Logical Reasoning](https://mlsanity.com/paper/2207.01450/) (2022)
- [Language Model Pre-training on True Negatives](https://mlsanity.com/paper/2212.00460/) (2022)
- [HYDRA -- Hyper Dependency Representation Attentions](https://mlsanity.com/paper/2109.05349/) (2021)
- [LySACoru~na at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing](https://mlsanity.com/paper/2204.12820/) (2022)
- [EvoText: Enhancing Natural Language Generation Models via Self-Escalation Learning for Up-to-Date Knowledge and Improved Performance](https://mlsanity.com/paper/2302.03896/) (2023)
- [GATology for Linguistics: What Syntactic Dependencies It Knows](https://mlsanity.com/paper/2305.13403/) (2023)
- [Syntactic Knowledge via Graph Attention with BERT in Machine Translation](https://mlsanity.com/paper/2305.13413/) (2023)

## Abstract (arXiv)

Understanding human language is one of the key themes of artificial intelligence. For language representation, the capacity of effectively modeling the linguistic knowledge from the detail-riddled and lengthy texts and getting rid of the noises is essential to improve its performance. Traditional attentive models attend to all words without explicit constraint, which results in inaccurate concentration on some dispensable words. In this work, we propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanisms for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation. The proposed SG-Net is applied to typical Transformer encoders. Extensive experiments on popular benchmark tasks, including machine reading comprehension, natural language inference, and neural machine translation show the effectiveness of the proposed SG-Net design.