# Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

arXiv: https://arxiv.org/abs/2204.11673 · HTML: https://mlsanity.com/paper/2204.11673/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 53 references that are in the map.
- First posted on arXiv 25 Apr 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation](https://mlsanity.com/paper/2010.02666/) (2020)
- [COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List](https://mlsanity.com/paper/2104.07186/) (2021)
- [StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://mlsanity.com/paper/1908.04577/) (2019)
- [CoLAKE: Contextualized Language and Knowledge Embedding](https://mlsanity.com/paper/2010.00309/) (2020)
- [Scalable Zero-shot Entity Linking with Dense Entity Retrieval](https://mlsanity.com/paper/1911.03814/) (2019)

## Built on by (most cited first, 10 of 10)

- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval](https://mlsanity.com/paper/2304.11370/) (2023)
- [THUIR@COLIEE 2023: Incorporating Structural Knowledge into Pre-trained Language Models for Legal Case Retrieval](https://mlsanity.com/paper/2305.06812/) (2023)
- [Constructing Tree-based Index for Efficient and Effective Dense Retrieval](https://mlsanity.com/paper/2304.11943/) (2023)
- [THUIR@COLIEE 2023: More Parameters and Legal Knowledge for Legal Case Entailment](https://mlsanity.com/paper/2305.06817/) (2023)
- [Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomy](https://mlsanity.com/paper/2403.04160/) (2024)
- [Graph Learning and Its Advancements on Large Language Models: A Holistic Survey](https://mlsanity.com/paper/2212.08966/) (2022)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [Entity Insertion in Multilingual Linked Corpora: The Case of Wikipedia](https://mlsanity.com/paper/2410.04254/) (2024)
- [Structures Facilitate Retrieve, Rerank, and Generate](https://mlsanity.com/paper/2606.03247/) (2026)

## Abstract (arXiv)

Passage re-ranking is to obtain a permutation over the candidate passage set from retrieval stage. Re-rankers have been boomed by Pre-trained Language Models (PLMs) due to their overwhelming advantages in natural language understanding. However, existing PLM based re-rankers may easily suffer from vocabulary mismatch and lack of domain specific knowledge. To alleviate these problems, explicit knowledge contained in knowledge graph is carefully introduced in our work. Specifically, we employ the existing knowledge graph which is incomplete and noisy, and first apply it in passage re-ranking task. To leverage a reliable knowledge, we propose a novel knowledge graph distillation method and obtain a knowledge meta graph as the bridge between query and passage. To align both kinds of embedding in the latent space, we employ PLM as text encoder and graph neural network over knowledge meta graph as knowledge encoder. Besides, a novel knowledge injector is designed for the dynamic interaction between text and knowledge encoder. Experimental results demonstrate the effectiveness of our method especially in queries requiring in-depth domain knowledge.