# Structural Adapters in Pretrained Language Models for AMR-to-text Generation

arXiv: https://arxiv.org/abs/2103.09120 · HTML: https://mlsanity.com/paper/2103.09120/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 65 references that are in the map.
- First posted on arXiv 16 Mar 2021.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [DART: Open-Domain Structured Data Record to Text Generation](https://mlsanity.com/paper/2007.02871/) (2020)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Graph Transformer for Graph-to-Sequence Learning](https://mlsanity.com/paper/1911.07470/) (2019)
- [Graph-to-Sequence Learning using Gated Graph Neural Networks](https://mlsanity.com/paper/1806.09835/) (2018)
- [Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers](https://mlsanity.com/paper/2005.11787/) (2020)

## Built on by (most cited first, 30 of 48)

- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Evaluating and Inducing Personality in Pre-trained Language Models](https://mlsanity.com/paper/2206.07550/) (2022)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [xGQA: Cross-Lingual Visual Question Answering](https://mlsanity.com/paper/2109.06082/) (2021)
- [Graph Pre-training for AMR Parsing and Generation](https://mlsanity.com/paper/2203.07836/) (2022)
- [Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance](https://mlsanity.com/paper/2402.08680/) (2024)
- [FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations](https://mlsanity.com/paper/2204.06508/) (2022)
- [Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction](https://mlsanity.com/paper/2403.18447/) (2024)
- [APPDIA: A Discourse-aware Transformer-based Style Transfer Model for Offensive Social Media Conversations](https://mlsanity.com/paper/2209.08207/) (2022)
- [Mixture of Soft Prompts for Controllable Data Generation](https://mlsanity.com/paper/2303.01580/) (2023)
- [SMATCH++: Standardized and Extended Evaluation of Semantic Graphs](https://mlsanity.com/paper/2305.06993/) (2023)
- [GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation](https://mlsanity.com/paper/2204.06674/) (2022)
- [Evaluating Generative Models for Graph-to-Text Generation](https://mlsanity.com/paper/2307.14712/) (2023)
- [LLM-based multi-agent poetry generation in non-cooperative environments](https://mlsanity.com/paper/2409.03659/) (2024)
- [Paraphrase Identification with Deep Learning: A Review of Datasets and Methods](https://mlsanity.com/paper/2212.06933/) (2022)
- [Unifying Structured Data as Graph for Data-to-Text Pre-Training](https://mlsanity.com/paper/2401.01183/) (2024)
- [Interpretable AMR-Based Question Decomposition for Multi-hop Question Answering](https://mlsanity.com/paper/2206.08486/) (2022)
- [AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters](https://mlsanity.com/paper/2108.08103/) (2021)
- [Self-supervised Graph Masking Pre-training for Graph-to-Text Generation](https://mlsanity.com/paper/2210.10599/) (2022)
- [Incorporating Graph Information in Transformer-based AMR Parsing](https://mlsanity.com/paper/2306.13467/) (2023)
- [Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs](https://mlsanity.com/paper/2406.04460/) (2024)
- [UKP-SQUARE: An Online Platform for Question Answering Research](https://mlsanity.com/paper/2203.13693/) (2022)
- [DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generation](https://mlsanity.com/paper/2212.08724/) (2022)
- [A Systematic Review of Data-to-Text NLG](https://mlsanity.com/paper/2402.08496/) (2024)
- [Smelting Gold and Silver for Improved Multilingual AMR-to-Text Generation](https://mlsanity.com/paper/2109.03808/) (2021)
- [Semantic-based Pre-training for Dialogue Understanding](https://mlsanity.com/paper/2209.09146/) (2022)
- [CATS: A Pragmatic Chinese Answer-to-Sequence Dataset with Large Scale and High Quality](https://mlsanity.com/paper/2306.11477/) (2023)
- [Text-Only Training for Visual Storytelling](https://mlsanity.com/paper/2308.08881/) (2023)
- [Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights](https://mlsanity.com/paper/2407.11449/) (2024)
- [Function-Guided Conditional Generation Using Protein Language Models with Adapters](https://mlsanity.com/paper/2410.03634/) (2024)

## Abstract (arXiv)

Pretrained language models (PLM) have recently advanced graph-to-text generation, where the input graph is linearized into a sequence and fed into the PLM to obtain its representation. However, efficiently encoding the graph structure in PLMs is challenging because such models were pretrained on natural language, and modeling structured data may lead to catastrophic forgetting of distributional knowledge. In this paper, we propose StructAdapt, an adapter method to encode graph structure into PLMs. Contrary to prior work, StructAdapt effectively models interactions among the nodes based on the graph connectivity, only training graph structure-aware adapter parameters. In this way, we incorporate task-specific knowledge while maintaining the topological structure of the graph. We empirically show the benefits of explicitly encoding graph structure into PLMs using StructAdapt, outperforming the state of the art on two AMR-to-text datasets, training only 5.1% of the PLM parameters.