# Graph Enhanced BERT for Query Understanding

arXiv: https://arxiv.org/abs/2204.06522 · HTML: https://mlsanity.com/paper/2204.06522/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 3 Apr 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Pre-training of Graph Augmented Transformers for Medication Recommendation](https://mlsanity.com/paper/1906.00346/) (2019)
- [Learning to Select Knowledge for Response Generation in Dialog Systems](https://mlsanity.com/paper/1902.04911/) (2019)
- [Unsupervised Pretraining for Sequence to Sequence Learning](https://mlsanity.com/paper/1611.02683/) (2016)
- [BERT-QE: Contextualized Query Expansion for Document Re-ranking](https://mlsanity.com/paper/2009.07258/) (2020)
- [Pre-trained Language Model based Ranking in Baidu Search](https://mlsanity.com/paper/2105.11108/) (2021)
- [A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networks](https://mlsanity.com/paper/1903.06464/) (2019)
- [User-Inspired Posterior Network for Recommendation Reason Generation](https://mlsanity.com/paper/2102.07919/) (2021)
- [Deep Search Query Intent Understanding](https://mlsanity.com/paper/2008.06759/) (2020)

## Built on by (most cited first, 3 of 3)

- [Learning To Rank Resources with GNN](https://mlsanity.com/paper/2304.07946/) (2023)
- [Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language Models](https://mlsanity.com/paper/2406.13556/) (2024)
- [REVISION:Reflective Intent Mining and Online Reasoning Auxiliary for E-commerce Visual Search System Optimization](https://mlsanity.com/paper/2510.22739/) (2025)

## Abstract (arXiv)

Query understanding plays a key role in exploring users' search intents and facilitating users to locate their most desired information. However, it is inherently challenging since it needs to capture semantic information from short and ambiguous queries and often requires massive task-specific labeled data. In recent years, pre-trained language models (PLMs) have advanced various natural language processing tasks because they can extract general semantic information from large-scale corpora. Therefore, there are unprecedented opportunities to adopt PLMs for query understanding. However, there is a gap between the goal of query understanding and existing pre-training strategies -- the goal of query understanding is to boost search performance while existing strategies rarely consider this goal. Thus, directly applying them to query understanding is sub-optimal. On the other hand, search logs contain user clicks between queries and urls that provide rich users' search behavioral information on queries beyond their content. Therefore, in this paper, we aim to fill this gap by exploring search logs. In particular, to incorporate search logs into pre-training, we first construct a query graph where nodes are queries and two queries are connected if they lead to clicks on the same urls. Then we propose a novel graph-enhanced pre-training framework, GE-BERT, which can leverage both query content and the query graph. In other words, GE-BERT can capture both the semantic information and the users' search behavioral information of queries. Extensive experiments on various query understanding tasks have demonstrated the effectiveness of the proposed framework.