# ExBERT: An External Knowledge Enhanced BERT for Natural Language Inference

arXiv: https://arxiv.org/abs/2108.01589 · HTML: https://mlsanity.com/paper/2108.01589/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 3 Aug 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling](https://mlsanity.com/paper/1906.07241/) (2019)
- [Improving Natural Language Inference Using External Knowledge in the Science Questions Domain](https://mlsanity.com/paper/1809.05724/) (2018)
- [Neural Natural Language Inference Models Enhanced with External Knowledge](https://mlsanity.com/paper/1711.04289/) (2017)
- [Leveraging Knowledge Bases in LSTMs for Improving Machine Reading](https://mlsanity.com/paper/1902.09091/) (2019)
- [AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples](https://mlsanity.com/paper/1805.04680/) (2018)
- [Infusing Knowledge into the Textual Entailment Task Using Graph Convolutional Networks](https://mlsanity.com/paper/1911.02060/) (2019)
- [Why Do Masked Neural Language Models Still Need Common Sense Knowledge?](https://mlsanity.com/paper/1911.03024/) (2019)
- [Improving Natural Language Inference with a Pretrained Parser](https://mlsanity.com/paper/1909.08217/) (2019)
- [Explicit Contextual Semantics for Text Comprehension](https://mlsanity.com/paper/1809.02794/) (2018)
- [Knowledge Enhanced Attention for Robust Natural Language Inference](https://mlsanity.com/paper/1909.00102/) (2019)
- [An Exploration of Dropout with RNNs for Natural Language Inference](https://mlsanity.com/paper/1810.08606/) (2018)
- [Bilinear Fusion of Commonsense Knowledge with Attention-Based NLI Models](https://mlsanity.com/paper/2010.11562/) (2020)

## Built on by (most cited first, 4 of 4)

- [ComFact: A Benchmark for Linking Contextual Commonsense Knowledge](https://mlsanity.com/paper/2210.12678/) (2022)
- [Exploration of Masked and Causal Language Modelling for Text Generation](https://mlsanity.com/paper/2405.12630/) (2024)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [Commonsense on Demand: Generating and Selectively Integrating Commonsense Knowledge for Natural Language Inference](https://mlsanity.com/paper/2507.15100/) (2025)

## Abstract (arXiv)

Neural language representation models such as BERT, pre-trained on large-scale unstructured corpora lack explicit grounding to real-world commonsense knowledge and are often unable to remember facts required for reasoning and inference. Natural Language Inference (NLI) is a challenging reasoning task that relies on common human understanding of language and real-world commonsense knowledge. We introduce a new model for NLI called External Knowledge Enhanced BERT (ExBERT), to enrich the contextual representation with real-world commonsense knowledge from external knowledge sources and enhance BERT's language understanding and reasoning capabilities. ExBERT takes full advantage of contextual word representations obtained from BERT and employs them to retrieve relevant external knowledge from knowledge graphs and to encode the retrieved external knowledge. Our model adaptively incorporates the external knowledge context required for reasoning over the inputs. Extensive experiments on the challenging SciTail and SNLI benchmarks demonstrate the effectiveness of ExBERT: in comparison to the previous state-of-the-art, we obtain an accuracy of 95.9% on SciTail and 91.5% on SNLI.