# Robust Retrieval Augmented Generation for Zero-shot Slot Filling

arXiv: https://arxiv.org/abs/2108.13934 · HTML: https://mlsanity.com/paper/2108.13934/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [Language Models are Open Knowledge Graphs](https://mlsanity.com/paper/2010.11967/) (2020)
- [Learning Dense Representations of Phrases at Scale](https://mlsanity.com/paper/2012.12624/) (2020)
- [Multi-task Retrieval for Knowledge-Intensive Tasks](https://mlsanity.com/paper/2101.00117/) (2021)
- [Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning](https://mlsanity.com/paper/1509.01692/) (2015)

## Built on by (most cited first, 24 of 24)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [Re2G: Retrieve, Rerank, Generate](https://mlsanity.com/paper/2207.06300/) (2022)
- [FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2209.14290/) (2022)
- [The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus](https://mlsanity.com/paper/2112.09924/) (2021)
- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2112.08688/) (2021)
- [Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximization](https://mlsanity.com/paper/2405.02816/) (2024)
- [RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing](https://mlsanity.com/paper/2404.19543/) (2024)
- [End-to-End Table Question Answering via Retrieval-Augmented Generation](https://mlsanity.com/paper/2203.16714/) (2022)
- [Applying a Generic Sequence-to-Sequence Model for Simple and Effective Keyphrase Generation](https://mlsanity.com/paper/2201.05302/) (2022)
- [KnowGL: Knowledge Generation and Linking from Text](https://mlsanity.com/paper/2210.13952/) (2022)
- [Multi-Row, Multi-Span Distant Supervision For Table+Text Question](https://mlsanity.com/paper/2112.07337/) (2021)
- [GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking](https://mlsanity.com/paper/2305.18144/) (2023)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling](https://mlsanity.com/paper/2207.03030/) (2022)
- [TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrieval](https://mlsanity.com/paper/2204.08173/) (2022)
- [Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learning](https://mlsanity.com/paper/2308.04712/) (2023)
- [Slot Filling for Biomedical Information Extraction](https://mlsanity.com/paper/2109.08564/) (2021)
- [KGI: An Integrated Framework for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2204.03985/) (2022)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [Retrieval-Based Transformer for Table Augmentation](https://mlsanity.com/paper/2306.11843/) (2023)
- [Cost-Efficient RAG for Entity Matching with LLMs: A Blocking-based Exploration](https://mlsanity.com/paper/2602.05708/) (2026)
- [Knowledge Graph Induction enabling Recommending and Trend Analysis: A Corporate Research Community Use Case](https://mlsanity.com/paper/2207.05188/) (2022)
- [Text Data Integration](https://mlsanity.com/paper/2603.27055/) (2026)

## Abstract (arXiv)

Automatically inducing high quality knowledge graphs from a given collection of documents still remains a challenging problem in AI. One way to make headway for this problem is through advancements in a related task known as slot filling. In this task, given an entity query in form of [Entity, Slot, ?], a system is asked to fill the slot by generating or extracting the missing value exploiting evidence extracted from relevant passage(s) in the given document collection. The recent works in the field try to solve this task in an end-to-end fashion using retrieval-based language models. In this paper, we present a novel approach to zero-shot slot filling that extends dense passage retrieval with hard negatives and robust training procedures for retrieval augmented generation models. Our model reports large improvements on both T-REx and zsRE slot filling datasets, improving both passage retrieval and slot value generation, and ranking at the top-1 position in the KILT leaderboard. Moreover, we demonstrate the robustness of our system showing its domain adaptation capability on a new variant of the TACRED dataset for slot filling, through a combination of zero/few-shot learning. We release the source code and pre-trained models.