# An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks

arXiv: https://arxiv.org/abs/2210.13002 · HTML: https://mlsanity.com/paper/2210.13002/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [Graph Pre-training for AMR Parsing and Generation](https://mlsanity.com/paper/2203.07836/) (2022)
- [SG-Net: Syntax-Guided Machine Reading Comprehension](https://mlsanity.com/paper/1908.05147/) (2019)
- [Do Syntax Trees Help Pre-trained Transformers Extract Information?](https://mlsanity.com/paper/2008.09084/) (2020)
- [Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa](https://mlsanity.com/paper/2104.04986/) (2021)
- [Tree-structured Attention with Hierarchical Accumulation](https://mlsanity.com/paper/2002.08046/) (2020)
- [Syntax-Enhanced Pre-trained Model](https://mlsanity.com/paper/2012.14116/) (2020)
- [Syntactic Structure Distillation Pretraining For Bidirectional Encoders](https://mlsanity.com/paper/2005.13482/) (2020)
- [Infusing Finetuning with Semantic Dependencies](https://mlsanity.com/paper/2012.05395/) (2020)
- [On Tree-Based Neural Sentence Modeling](https://mlsanity.com/paper/1808.09644/) (2018)
- [Learning to Deceive Knowledge Graph Augmented Models via Targeted Perturbation](https://mlsanity.com/paper/2010.12872/) (2020)
- [Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling](https://mlsanity.com/paper/2112.07874/) (2021)
- [Bird's Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach](https://mlsanity.com/paper/2105.02629/) (2021)
- [Does Knowledge Help General NLU? An Empirical Study](https://mlsanity.com/paper/2109.00563/) (2021)

## Built on by (most cited first, 2 of 2)

- [Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models](https://mlsanity.com/paper/2310.16240/) (2023)
- [Trustful LLMs: Customizing and Grounding Text Generation with Knowledge Bases and Dual Decoders](https://mlsanity.com/paper/2411.07870/) (2024)

## Abstract (arXiv)

Though linguistic knowledge emerges during large-scale language model pretraining, recent work attempt to explicitly incorporate human-defined linguistic priors into task-specific fine-tuning. Infusing language models with syntactic or semantic knowledge from parsers has shown improvements on many language understanding tasks. To further investigate the effectiveness of structural linguistic priors, we conduct empirical study of replacing parsed graphs or trees with trivial ones (rarely carrying linguistic knowledge e.g., balanced tree) for tasks in the GLUE benchmark. Encoding with trivial graphs achieves competitive or even better performance in fully-supervised and few-shot settings. It reveals that the gains might not be significantly attributed to explicit linguistic priors but rather to more feature interactions brought by fusion layers. Hence we call for attention to using trivial graphs as necessary baselines to design advanced knowledge fusion methods in the future.