# Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling

arXiv: https://arxiv.org/abs/2104.07704 · HTML: https://mlsanity.com/paper/2104.07704/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 76 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling](https://mlsanity.com/paper/1703.04826/) (2017)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [Improve Transformer Models with Better Relative Position Embeddings](https://mlsanity.com/paper/2009.13658/) (2020)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [Head-Driven Phrase Structure Grammar Parsing on Penn Treebank](https://mlsanity.com/paper/1907.02684/) (2019)
- [Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling](https://mlsanity.com/paper/1805.04787/) (2018)
- [Deep Semantic Role Labeling with Self-Attention](https://mlsanity.com/paper/1712.01586/) (2017)
- [Dependency or Span, End-to-End Uniform Semantic Role Labeling](https://mlsanity.com/paper/1901.05280/) (2019)
- [Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement](https://mlsanity.com/paper/2003.13118/) (2020)
- [Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification](https://mlsanity.com/paper/1704.00217/) (2017)
- [Neural Semantic Role Labeling with Dependency Path Embeddings](https://mlsanity.com/paper/1605.07515/) (2016)
- [A Span Selection Model for Semantic Role Labeling](https://mlsanity.com/paper/1810.02245/) (2018)
- [A Full End-to-End Semantic Role Labeler, Syntax-agnostic Over Syntax-aware?](https://mlsanity.com/paper/1808.03815/) (2018)
- [Parsing All: Syntax and Semantics, Dependencies and Spans](https://mlsanity.com/paper/1908.11522/) (2019)
- [Graph-to-Graph Transformer for Transition-based Dependency Parsing](https://mlsanity.com/paper/1911.03561/) (2019)
- [A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling](https://mlsanity.com/paper/1701.02593/) (2017)
- [Syntax-aware Neural Semantic Role Labeling](https://mlsanity.com/paper/1907.09312/) (2019)
- [Syntax-aware Multilingual Semantic Role Labeling](https://mlsanity.com/paper/1909.00310/) (2019)
- [Syntax Role for Neural Semantic Role Labeling](https://mlsanity.com/paper/2009.05737/) (2020)
- [Syntax-aware Neural Semantic Role Labeling with Supertags](https://mlsanity.com/paper/1903.05260/) (2019)
- [Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task](https://mlsanity.com/paper/1910.03291/) (2019)
- [Improved Semantic Role Labeling using Parameterized Neighborhood Memory Adaptation](https://mlsanity.com/paper/2011.14459/) (2020)
- [Semantic Role Labeling with Iterative Structure Refinement](https://mlsanity.com/paper/1909.03285/) (2019)

## Built on by (most cited first, 8 of 8)

- [RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question](https://mlsanity.com/paper/2211.01482/) (2022)
- [Semantic Role Labeling as Dependency Parsing: Exploring Latent Tree Structures Inside Arguments](https://mlsanity.com/paper/2110.06865/) (2021)
- [Ensemble Distillation for Unsupervised Constituency Parsing](https://mlsanity.com/paper/2310.01717/) (2023)
- [Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders](https://mlsanity.com/paper/2311.08579/) (2023)
- [Transformers as Graph-to-Graph Models](https://mlsanity.com/paper/2310.17936/) (2023)
- [An MRC Framework for Semantic Role Labeling](https://mlsanity.com/paper/2109.06660/) (2021)
- [Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers](https://mlsanity.com/paper/2205.11456/) (2022)
- [GADePo: Graph-Assisted Declarative Pooling Transformers for Document-Level Relation Extraction](https://mlsanity.com/paper/2308.14423/) (2023)

## Abstract (arXiv)

Recent models have shown that incorporating syntactic knowledge into the semantic role labelling (SRL) task leads to a significant improvement. In this paper, we propose Syntax-aware Graph-to-Graph Transformer (SynG2G-Tr) model, which encodes the syntactic structure using a novel way to input graph relations as embeddings, directly into the self-attention mechanism of Transformer. This approach adds a soft bias towards attention patterns that follow the syntactic structure but also allows the model to use this information to learn alternative patterns. We evaluate our model on both span-based and dependency-based SRL datasets, and outperform previous alternative methods in both in-domain and out-of-domain settings, on CoNLL 2005 and CoNLL 2009 datasets.