# Multi-task Retrieval for Knowledge-Intensive Tasks

arXiv: https://arxiv.org/abs/2101.00117 · HTML: https://mlsanity.com/paper/2101.00117/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 1 Jan 2021.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [CERT: Contrastive Self-supervised Learning for Language Understanding](https://mlsanity.com/paper/2005.12766/) (2020)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)](https://mlsanity.com/paper/1405.5869/) (2014)
- [Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering](https://mlsanity.com/paper/1908.08167/) (2019)
- [Relevance-guided Supervision for OpenQA with ColBERT](https://mlsanity.com/paper/2007.00814/) (2020)
- [Quantization based Fast Inner Product Search](https://mlsanity.com/paper/1509.01469/) (2015)

## Built on by (most cited first, 30 of 44)

- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [Task-aware Retrieval with Instructions](https://mlsanity.com/paper/2211.09260/) (2022)
- [Re2G: Retrieve, Rerank, Generate](https://mlsanity.com/paper/2207.06300/) (2022)
- [FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2209.14290/) (2022)
- [Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?](https://mlsanity.com/paper/2110.06918/) (2021)
- [The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus](https://mlsanity.com/paper/2112.09924/) (2021)
- [Domain-matched Pre-training Tasks for Dense Retrieval](https://mlsanity.com/paper/2107.13602/) (2021)
- [Chain-of-Retrieval Augmented Generation](https://mlsanity.com/paper/2501.14342/) (2025)
- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2112.08688/) (2021)
- [FaVIQ: FAct Verification from Information-seeking Questions](https://mlsanity.com/paper/2107.02153/) (2021)
- [CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks](https://mlsanity.com/paper/2208.07652/) (2022)
- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [Robust Retrieval Augmented Generation for Zero-shot Slot Filling](https://mlsanity.com/paper/2108.13934/) (2021)
- [CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks](https://mlsanity.com/paper/2402.01176/) (2024)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Improving Wikipedia Verifiability with AI](https://mlsanity.com/paper/2207.06220/) (2022)
- [GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking](https://mlsanity.com/paper/2305.18144/) (2023)
- [Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning](https://mlsanity.com/paper/2204.02725/) (2022)
- [Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling](https://mlsanity.com/paper/2207.03030/) (2022)
- [TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrieval](https://mlsanity.com/paper/2204.08173/) (2022)
- [The Web Can Be Your Oyster for Improving Large Language Models](https://mlsanity.com/paper/2305.10998/) (2023)
- [Nonparametric Decoding for Generative Retrieval](https://mlsanity.com/paper/2210.02068/) (2022)
- [Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models](https://mlsanity.com/paper/2211.00915/) (2022)
- [Data-Efficient Autoregressive Document Retrieval for Fact Verification](https://mlsanity.com/paper/2211.09388/) (2022)
- [RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets](https://mlsanity.com/paper/2511.01386/) (2025)
- [Artefact Retrieval: Overview of NLP Models with Knowledge Base Access](https://mlsanity.com/paper/2201.09651/) (2022)
- [Slot Filling for Biomedical Information Extraction](https://mlsanity.com/paper/2109.08564/) (2021)
- [KGI: An Integrated Framework for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2204.03985/) (2022)

## Abstract (arXiv)

Retrieving relevant contexts from a large corpus is a crucial step for tasks such as open-domain question answering and fact checking. Although neural retrieval outperforms traditional methods like tf-idf and BM25, its performance degrades considerably when applied to out-of-domain data. Driven by the question of whether a neural retrieval model can be universal and perform robustly on a wide variety of problems, we propose a multi-task trained model. Our approach not only outperforms previous methods in the few-shot setting, but also rivals specialised neural retrievers, even when in-domain training data is abundant. With the help of our retriever, we improve existing models for downstream tasks and closely match or improve the state of the art on multiple benchmarks.