# Retrofitting Structure-aware Transformer Language Model for End Tasks

arXiv: https://arxiv.org/abs/2009.07408 · HTML: https://mlsanity.com/paper/2009.07408/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2020.

## Built on by (most cited first, 24 of 24)

- [Reasoning Implicit Sentiment with Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.11255/) (2023)
- [LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model](https://mlsanity.com/paper/2304.06248/) (2023)
- [Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination](https://mlsanity.com/paper/2305.12256/) (2023)
- [On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training](https://mlsanity.com/paper/2304.09563/) (2023)
- [Generating Visual Spatial Description via Holistic 3D Scene Understanding](https://mlsanity.com/paper/2305.11768/) (2023)
- [Modeling Hierarchical Structures with Continuous Recursive Neural Networks](https://mlsanity.com/paper/2106.06038/) (2021)
- [Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment](https://mlsanity.com/paper/2305.12260/) (2023)
- [Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling](https://mlsanity.com/paper/2308.05081/) (2023)
- [Conversational Semantic Role Labeling with Predicate-Oriented Latent Graph](https://mlsanity.com/paper/2210.03037/) (2022)
- [Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognition](https://mlsanity.com/paper/2308.04502/) (2023)
- [Constructing Code-mixed Universal Dependency Forest for Unbiased Cross-lingual Relation Extraction](https://mlsanity.com/paper/2305.12258/) (2023)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [Cooperative Self-training of Machine Reading Comprehension](https://mlsanity.com/paper/2103.07449/) (2021)
- [Efficient Beam Tree Recursion](https://mlsanity.com/paper/2307.10779/) (2023)
- [Learning Shortcuts: On the Misleading Promise of NLU in Language Models](https://mlsanity.com/paper/2401.09615/) (2024)
- [ECQED: Emotion-Cause Quadruple Extraction in Dialogs](https://mlsanity.com/paper/2306.03969/) (2023)
- [TKDP: Threefold Knowledge-enriched Deep Prompt Tuning for Few-shot Named Entity Recognition](https://mlsanity.com/paper/2306.03974/) (2023)
- [Revisiting Conversation Discourse for Dialogue Disentanglement](https://mlsanity.com/paper/2306.03975/) (2023)
- [A Bi-directional Multi-hop Inference Model for Joint Dialog Sentiment Classification and Act Recognition](https://mlsanity.com/paper/2308.04424/) (2023)
- [Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning](https://mlsanity.com/paper/2310.18930/) (2023)
- [DialogRE^C+: An Extension of DialogRE to Investigate How Much Coreference Helps Relation Extraction in Dialogs](https://mlsanity.com/paper/2308.04498/) (2023)
- [Syntactic Fusion: Enhancing Aspect-Level Sentiment Analysis Through Multi-Tree Graph Integration](https://mlsanity.com/paper/2312.03738/) (2023)
- [Dynamic Syntax Mapping: A New Approach to Unsupervised Syntax Parsing](https://mlsanity.com/paper/2312.14966/) (2023)
- [Grammar Induction from Visual, Speech and Text](https://mlsanity.com/paper/2410.03739/) (2024)

## Abstract (arXiv)

We consider retrofitting structure-aware Transformer-based language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme. Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases. By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks.