# Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusion

arXiv: https://arxiv.org/abs/2210.08471 · HTML: https://mlsanity.com/paper/2210.08471/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 16 Oct 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [Breaking the Softmax Bottleneck: A High-Rank RNN Language Model](https://mlsanity.com/paper/1711.03953/) (2017)
- [Self-Attention Attribution: Interpreting Information Interactions Inside Transformer](https://mlsanity.com/paper/2004.11207/) (2020)
- [Bilateral Multi-Perspective Matching for Natural Language Sentences](https://mlsanity.com/paper/1702.03814/) (2017)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [A Continuously Growing Dataset of Sentential Paraphrases](https://mlsanity.com/paper/1708.00391/) (2017)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees](https://mlsanity.com/paper/2103.04350/) (2021)
- [Learning to Compose Task-Specific Tree Structures](https://mlsanity.com/paper/1707.02786/) (2017)
- [Do Syntax Trees Help Pre-trained Transformers Extract Information?](https://mlsanity.com/paper/2008.09084/) (2020)
- [A Fast Unified Model for Parsing and Sentence Understanding](https://mlsanity.com/paper/1603.06021/) (2016)
- [A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations](https://mlsanity.com/paper/1511.08277/) (2015)
- [Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks](https://mlsanity.com/paper/2102.10934/) (2021)
- [DABERT: Dual Attention Enhanced BERT for Semantic Matching](https://mlsanity.com/paper/2210.03454/) (2022)
- [Scheduled Multi-Task Learning: From Syntax to Translation](https://mlsanity.com/paper/1804.08915/) (2018)
- [Co-Stack Residual Affinity Networks with Multi-level Attention Refinement for Matching Text Sequences](https://mlsanity.com/paper/1810.02938/) (2018)

## Built on by (most cited first, 18 of 18)

- [Time-aware Multiway Adaptive Fusion Network for Temporal Knowledge Graph Question Answering](https://mlsanity.com/paper/2302.12529/) (2023)
- [Question Calibration and Multi-Hop Modeling for Temporal Question Answering](https://mlsanity.com/paper/2402.13188/) (2024)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflicts](https://mlsanity.com/paper/2302.12530/) (2023)
- [Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance](https://mlsanity.com/paper/2508.21741/) (2025)
- [Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling](https://mlsanity.com/paper/2509.25361/) (2025)
- [Comateformer: Combined Attention Transformer for Semantic Sentence Matching](https://mlsanity.com/paper/2412.07220/) (2024)
- [StructCoh: Structured Contrastive Learning for Context-Aware Text Semantic Matching](https://mlsanity.com/paper/2509.02033/) (2025)
- [Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models](https://mlsanity.com/paper/2604.10079/) (2026)
- [Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty](https://mlsanity.com/paper/2604.10072/) (2026)
- [DPI: Exploiting Parameter Heterogeneity for Interference-Free Fine-Tuning](https://mlsanity.com/paper/2601.17777/) (2026)
- [Reinforcement Learning Enhanced Multi-hop Reasoning for Temporal Knowledge Question Answering](https://mlsanity.com/paper/2601.01195/) (2026)
- [Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning](https://mlsanity.com/paper/2604.14010/) (2026)
- [Using External knowledge to Enhanced PLM for Semantic Matching](https://mlsanity.com/paper/2505.06605/) (2025)
- [Boosting Neural Language Inference via Cascaded Interactive Reasoning](https://mlsanity.com/paper/2505.06607/) (2025)
- [Temporal-Aware Heterogeneous Graph Reasoning with Multi-View Fusion for Temporal Question Answering](https://mlsanity.com/paper/2602.19569/) (2026)
- [Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization](https://mlsanity.com/paper/2604.17051/) (2026)
- [Multi-Granularity Reasoning for Natural Language Inference](https://mlsanity.com/paper/2606.05181/) (2026)

## Abstract (arXiv)

Transformer-based pre-trained models like BERT have achieved great progress on Semantic Sentence Matching. Meanwhile, dependency prior knowledge has also shown general benefits in multiple NLP tasks. However, how to efficiently integrate dependency prior structure into pre-trained models to better model complex semantic matching relations is still unsettled. In this paper, we propose the Dependency-Enhanced Adaptive Fusion Attention (DAFA), which explicitly introduces dependency structure into pre-trained models and adaptively fuses it with semantic information. Specifically, textbf(i) DAFA first proposes a structure-sensitive paradigm to construct a dependency matrix for calibrating attention weights. It adopts an adaptive fusion module to integrate the obtained dependency information and the original semantic signals. Moreover, DAFA reconstructs the attention calculation flow and provides better interpretability. By applying it on BERT, our method achieves state-of-the-art or competitive performance on 10 public datasets, demonstrating the benefits of adaptively fusing dependency structure in semantic matching task.