# Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing

arXiv: https://arxiv.org/abs/2110.15534 · HTML: https://mlsanity.com/paper/2110.15534/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 29 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2010.03546/) (2020)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [Grammar as a Foreign Language](https://mlsanity.com/paper/1412.7449/) (2014)
- [Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training](https://mlsanity.com/paper/2012.10309/) (2020)
- [Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2001.11458/) (2020)
- [Leveraging Abstract Meaning Representation for Knowledge Base Question Answering](https://mlsanity.com/paper/2012.01707/) (2020)
- [AMR Parsing as Sequence-to-Graph Transduction](https://mlsanity.com/paper/1905.08704/) (2019)
- [AMR Parsing via Graph-Sequence Iterative Inference](https://mlsanity.com/paper/2004.05572/) (2020)
- [Toward Abstractive Summarization Using Semantic Representations](https://mlsanity.com/paper/1805.10399/) (2018)
- [Transition-based Parsing with Stack-Transformers](https://mlsanity.com/paper/2010.10669/) (2020)
- [AMR Parsing as Graph Prediction with Latent Alignment](https://mlsanity.com/paper/1805.05286/) (2018)
- [An Incremental Parser for Abstract Meaning Representation](https://mlsanity.com/paper/1608.06111/) (2016)
- [Broad-Coverage Semantic Parsing as Transduction](https://mlsanity.com/paper/1909.02607/) (2019)
- [Improving AMR Parsing with Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2010.01771/) (2020)
- [Rewarding Smatch: Transition-Based AMR Parsing with Reinforcement Learning](https://mlsanity.com/paper/1905.13370/) (2019)
- [AMR Parsing with Action-Pointer Transformer](https://mlsanity.com/paper/2104.14674/) (2021)
- [Structural Guidance for Transformer Language Models](https://mlsanity.com/paper/2108.00104/) (2021)
- [Guided Neural Language Generation for Abstractive Summarization using Abstract Meaning Representation](https://mlsanity.com/paper/1808.09160/) (2018)
- [Neural Semantic Parsing by Character-based Translation: Experiments with Abstract Meaning Representations](https://mlsanity.com/paper/1705.09980/) (2017)
- [Pushing the Limits of AMR Parsing with Self-Learning](https://mlsanity.com/paper/2010.10673/) (2020)
- [AMR Parsing using Stack-LSTMs](https://mlsanity.com/paper/1707.07755/) (2017)
- [Robust Incremental Neural Semantic Graph Parsing](https://mlsanity.com/paper/1704.07092/) (2017)
- [Encoder-Decoder Shift-Reduce Syntactic Parsing](https://mlsanity.com/paper/1706.07905/) (2017)
- [A Differentiable Relaxation of Graph Segmentation and Alignment for AMR Parsing](https://mlsanity.com/paper/2010.12676/) (2020)
- [A Hierarchy of Graph Neural Networks Based on Learnable Local Features](https://mlsanity.com/paper/1911.05256/) (2019)

## Built on by (most cited first, 25 of 25)

- [Prompting Large Language Models With the Socratic Method](https://mlsanity.com/paper/2303.08769/) (2023)
- [Maximum Bayes Smatch Ensemble Distillation for AMR Parsing](https://mlsanity.com/paper/2112.07790/) (2021)
- [KnowGL: Knowledge Generation and Linking from Text](https://mlsanity.com/paper/2210.13952/) (2022)
- [DocAMR: Multi-Sentence AMR Representation and Evaluation](https://mlsanity.com/paper/2112.08513/) (2021)
- [Inducing and Using Alignments for Transition-based AMR Parsing](https://mlsanity.com/paper/2205.01464/) (2022)
- [ATP: AMRize Then Parse! Enhancing AMR Parsing with PseudoAMRs](https://mlsanity.com/paper/2204.08875/) (2022)
- [Learning to Transpile AMR into SPARQL](https://mlsanity.com/paper/2112.07877/) (2021)
- [Utterance Classification with Logical Neural Network: Explainable AI for Mental Disorder Diagnosis](https://mlsanity.com/paper/2306.03902/) (2023)
- [A Comparative Study of Neurosymbolic AI Approaches to Interpretable Logical Reasoning](https://mlsanity.com/paper/2508.03366/) (2025)
- [AMR Parsing with Instruction Fine-tuned Pre-trained Language Models](https://mlsanity.com/paper/2304.12272/) (2023)
- [Cross-domain Generalization for AMR Parsing](https://mlsanity.com/paper/2210.12445/) (2022)
- [Laziness Is a Virtue When It Comes to Compositionality in Neural Semantic Parsing](https://mlsanity.com/paper/2305.04346/) (2023)
- [NetFlowGen: Leveraging Generative Pre-training for Network Traffic Dynamics](https://mlsanity.com/paper/2412.20635/) (2024)
- [Survey of Abstract Meaning Representation: Then, Now, Future](https://mlsanity.com/paper/2505.03229/) (2025)
- [AMR Parsing with Causal Hierarchical Attention and Pointers](https://mlsanity.com/paper/2310.11964/) (2023)
- [Modeling Emotions and Ethics with Large Language Models](https://mlsanity.com/paper/2404.13071/) (2024)
- [Pairwise Logical Selection of Enthymeme Completions under Semantic-Link Uncertainty](https://mlsanity.com/paper/2608.18820/) (2026)
- [Cross-lingual AMR Aligner: Paying Attention to Cross-Attention](https://mlsanity.com/paper/2206.07587/) (2022)
- [DiffG-RL: Leveraging Difference between State and Common Sense](https://mlsanity.com/paper/2211.16002/) (2022)
- [Slide, Constrain, Parse, Repeat: Synchronous SlidingWindows for Document AMR Parsing](https://mlsanity.com/paper/2305.17273/) (2023)
- [Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency](https://mlsanity.com/paper/2305.20018/) (2023)
- [TESS: A Multi-intent Parser for Conversational Multi-Agent Systems with Decentralized Natural Language Understanding Models](https://mlsanity.com/paper/2312.11828/) (2023)
- [FLAG: Financial Long Document Classification via AMR-based GNN](https://mlsanity.com/paper/2410.02024/) (2024)
- [Train Once for All: A Transitional Approach for Efficient Aspect Sentiment Triplet Extraction](https://mlsanity.com/paper/2412.00208/) (2024)
- [SETUP: Sentence-level English-To-Uniform Meaning Representation Parser](https://mlsanity.com/paper/2512.07068/) (2025)

## Abstract (arXiv)

Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization.