# Graph Pre-training for AMR Parsing and Generation

arXiv: https://arxiv.org/abs/2203.07836 · HTML: https://mlsanity.com/paper/2203.07836/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 64 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Graph-to-Sequence Learning using Gated Graph Neural Networks](https://mlsanity.com/paper/1806.09835/) (2018)
- [Neural AMR: Sequence-to-Sequence Models for Parsing and Generation](https://mlsanity.com/paper/1704.08381/) (2017)
- [A Graph-to-Sequence Model for AMR-to-Text Generation](https://mlsanity.com/paper/1805.02473/) (2018)
- [AMR Parsing as Sequence-to-Graph Transduction](https://mlsanity.com/paper/1905.08704/) (2019)
- [Structural Adapters in Pretrained Language Models for AMR-to-text Generation](https://mlsanity.com/paper/2103.09120/) (2021)
- [AMR Parsing via Graph-Sequence Iterative Inference](https://mlsanity.com/paper/2004.05572/) (2020)
- [Toward Abstractive Summarization Using Semantic Representations](https://mlsanity.com/paper/1805.10399/) (2018)
- [GPT-too: A language-model-first approach for AMR-to-text generation](https://mlsanity.com/paper/2005.09123/) (2020)
- [Transition-based Parsing with Stack-Transformers](https://mlsanity.com/paper/2010.10669/) (2020)
- [Semantic Neural Machine Translation using AMR](https://mlsanity.com/paper/1902.07282/) (2019)
- [AMR Parsing as Graph Prediction with Latent Alignment](https://mlsanity.com/paper/1805.05286/) (2018)
- [Abstract Meaning Representation for Multi-Document Summarization](https://mlsanity.com/paper/1806.05655/) (2018)
- [Modeling Graph Structure in Transformer for Better AMR-to-Text Generation](https://mlsanity.com/paper/1909.00136/) (2019)
- [Broad-Coverage Semantic Parsing as Transduction](https://mlsanity.com/paper/1909.02607/) (2019)
- [Improving AMR Parsing with Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2010.01771/) (2020)
- [AMR Parsing with Action-Pointer Transformer](https://mlsanity.com/paper/2104.14674/) (2021)
- [Semantic Representation for Dialogue Modeling](https://mlsanity.com/paper/2105.10188/) (2021)
- [Structural Neural Encoders for AMR-to-text Generation](https://mlsanity.com/paper/1903.11410/) (2019)

## Built on by (most cited first, 30 of 55)

- [FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing](https://mlsanity.com/paper/2305.17497/) (2023)
- [Don't Forget to Connect! Improving RAG with Graph-based Reranking](https://mlsanity.com/paper/2405.18414/) (2024)
- [FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval](https://mlsanity.com/paper/2503.21309/) (2025)
- [AMPERE: AMR-Aware Prefix for Generation-Based Event Argument Extraction Model](https://mlsanity.com/paper/2305.16734/) (2023)
- [ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget](https://mlsanity.com/paper/2408.00103/) (2024)
- ["You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation](https://mlsanity.com/paper/2310.17793/) (2023)
- [BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing](https://mlsanity.com/paper/2206.10668/) (2022)
- [Unifying Structured Data as Graph for Data-to-Text Pre-Training](https://mlsanity.com/paper/2401.01183/) (2024)
- [Inducing and Using Alignments for Transition-based AMR Parsing](https://mlsanity.com/paper/2205.01464/) (2022)
- [Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation](https://mlsanity.com/paper/2306.00124/) (2023)
- [An AMR-based Link Prediction Approach for Document-level Event Argument Extraction](https://mlsanity.com/paper/2305.19162/) (2023)
- [ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions](https://mlsanity.com/paper/2406.04286/) (2024)
- [Incorporating Graph Information in Transformer-based AMR Parsing](https://mlsanity.com/paper/2306.13467/) (2023)
- [T-STAR: Truthful Style Transfer using AMR Graph as Intermediate Representation](https://mlsanity.com/paper/2212.01667/) (2022)
- [AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite](https://mlsanity.com/paper/2312.03480/) (2023)
- [Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations](https://mlsanity.com/paper/2404.12698/) (2024)
- [Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation](https://mlsanity.com/paper/2210.09773/) (2022)
- [Entailment-Preserving First-order Logic Representations in Natural Language Entailment](https://mlsanity.com/paper/2502.16757/) (2025)
- [Causal Document-Grounded Dialogue Pre-training](https://mlsanity.com/paper/2305.10927/) (2023)
- [A Systematic Review of Data-to-Text NLG](https://mlsanity.com/paper/2402.08496/) (2024)
- [Semantic Graphs for Syntactic Simplification: A Revisit from the Age of LLM](https://mlsanity.com/paper/2407.04067/) (2024)
- [Semantic-based Pre-training for Dialogue Understanding](https://mlsanity.com/paper/2209.09146/) (2022)
- [AMR Parsing with Instruction Fine-tuned Pre-trained Language Models](https://mlsanity.com/paper/2304.12272/) (2023)
- [AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing](https://mlsanity.com/paper/2306.10786/) (2023)
- [Prompting for Numerical Sequences: A Case Study on Market Comment Generation](https://mlsanity.com/paper/2404.02466/) (2024)
- [MACT: Model-Agnostic Cross-Lingual Training for Discourse Representation Structure Parsing](https://mlsanity.com/paper/2406.01052/) (2024)
- [SR-LLM: Rethinking the Structured Representation in Large Language Model](https://mlsanity.com/paper/2502.14352/) (2025)
- [Cross-domain Generalization for AMR Parsing](https://mlsanity.com/paper/2210.12445/) (2022)
- [An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks](https://mlsanity.com/paper/2210.13002/) (2022)
- [Exploiting Abstract Meaning Representation for Open-Domain Question Answering](https://mlsanity.com/paper/2305.17050/) (2023)

## Abstract (arXiv)

Abstract meaning representation (AMR) highlights the core semantic information of text in a graph structure. Recently, pre-trained language models (PLMs) have advanced tasks of AMR parsing and AMR-to-text generation, respectively. However, PLMs are typically pre-trained on textual data, thus are sub-optimal for modeling structural knowledge. To this end, we investigate graph self-supervised training to improve the structure awareness of PLMs over AMR graphs. In particular, we introduce two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-training. We further design a unified framework to bridge the gap between pre-training and fine-tuning tasks. Experiments on both AMR parsing and AMR-to-text generation show the superiority of our model. To our knowledge, we are the first to consider pre-training on semantic graphs.