# Contextualized Representations Using Textual Encyclopedic Knowledge

arXiv: https://arxiv.org/abs/2004.12006 · HTML: https://mlsanity.com/paper/2004.12006/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 17 of 17)

- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [Rethinking with Retrieval: Faithful Large Language Model Inference](https://mlsanity.com/paper/2301.00303/) (2022)
- [BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models](https://mlsanity.com/paper/2403.18365/) (2024)
- [Synergistic Interplay between Search and Large Language Models for Information Retrieval](https://mlsanity.com/paper/2305.07402/) (2023)
- [Modeling Context in Answer Sentence Selection Systems on a Latency Budget](https://mlsanity.com/paper/2101.12093/) (2021)
- [Elaboration-Generating Commonsense Question Answering at Scale](https://mlsanity.com/paper/2209.01232/) (2022)
- [KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding](https://mlsanity.com/paper/2210.04105/) (2022)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)
- [Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Induction](https://mlsanity.com/paper/2202.04876/) (2022)
- [Comprehension Based Question Answering using Bloom's Taxonomy](https://mlsanity.com/paper/2106.04653/) (2021)
- [A Survey of Knowledge Enhanced Pre-trained Models](https://mlsanity.com/paper/2110.00269/) (2021)
- [3D-EX : A Unified Dataset of Definitions and Dictionary Examples](https://mlsanity.com/paper/2308.03043/) (2023)
- [Go Forth and Prosper: Language Modeling with Ancient Textual History](https://mlsanity.com/paper/2104.08742/) (2021)
- [Metadata Shaping: Natural Language Annotations for the Tail](https://mlsanity.com/paper/2110.08430/) (2021)
- [UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models](https://mlsanity.com/paper/2305.01624/) (2023)
- [WIKITIDE: A Wikipedia-Based Timestamped Definition Pairs Dataset](https://mlsanity.com/paper/2308.03582/) (2023)
- [Robo-CSK-Organizer: Commonsense Knowledge to Organize Detected Objects for Multipurpose Robots](https://mlsanity.com/paper/2409.18385/) (2024)

## Abstract (arXiv)

We present a method to represent input texts by contextualizing them jointly with dynamically retrieved textual encyclopedic background knowledge from multiple documents. We apply our method to reading comprehension tasks by encoding questions and passages together with background sentences about the entities they mention. We show that integrating background knowledge from text is effective for tasks focusing on factual reasoning and allows direct reuse of powerful pretrained BERT-style encoders. Moreover, knowledge integration can be further improved with suitable pretraining via a self-supervised masked language model objective over words in background-augmented input text. On TriviaQA, our approach obtains improvements of 1.6 to 3.1 F1 over comparable RoBERTa models which do not integrate background knowledge dynamically. On MRQA, a large collection of diverse QA datasets, we see consistent gains in-domain along with large improvements out-of-domain on BioASQ (2.1 to 4.2 F1), TextbookQA (1.6 to 2.0 F1), and DuoRC (1.1 to 2.0 F1).