mlsanityData to 7 Oct 2026

Multi-task Retrieval for Knowledge-Intensive Tasks

arXiv 2101.00117 ↗First posted 1 Jan 2021cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

Retrieving relevant contexts from a large corpus is a crucial step for tasks such as open-domain question answering and fact checking. Although neural retrieval outperforms traditional methods like tf-idf and BM25, its performance degrades considerably when applied to out-of-domain data. Driven by the question of whether a neural retrieval model can be universal and perform robustly on a wide variety of problems, we propose a multi-task trained model. Our approach not only outperforms previous methods in the few-shot setting, but also rivals specialised neural retrievers, even when in-domain training data is abundant. With the help of our retriever, we improve existing models for downstream tasks and closely match or improve the state of the art on multiple benchmarks.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 1 Jan 2021.

Builds on

27 of 43 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
15 more
  1. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  2. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  3. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  4. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  5. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  6. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  7. 2020Autoregressive Entity Retrievalcited by 333
  8. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  9. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  10. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  11. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  12. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  13. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  14. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  15. 2015Quantization based Fast Inner Product Searchcited by 29

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 20; 2023: 6; 2024: 3; 2025: 5.

44 papers.

Sort:

By area:

  1. 2021A Survey on Automated Fact-Checkingcited by 376
  2. 2022A Review on Language Models as Knowledge Basescited by 156
  3. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  4. 2022Task-aware Retrieval with Instructionscited by 107
  5. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  6. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  7. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  8. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  9. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  10. 2025Chain-of-Retrieval Augmented Generationcited by 48
  11. 2021Evidentiality-guided Generation for Knowledge-Intensive NLP Taskscited by 40
  12. 2021FaVIQ: FAct Verification from Information-seeking Questionscited by 36
32 more
  1. 2022CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Taskscited by 36
  2. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  3. 2021Robust Retrieval Augmented Generation for Zero-shot Slot Fillingcited by 24
  4. 2024CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Taskscited by 18
  5. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  6. 2022Improving Wikipedia Verifiability with AIcited by 11
  7. 2023GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Rankingcited by 11
  8. 2022Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learningcited by 10
  9. 2022Multi-Task Retrieval-Augmented Text Generation with Relevance Samplingcited by 10
  10. 2022TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrievalcited by 8
  11. 2023The Web Can Be Your Oyster for Improving Large Language Modelscited by 7
  12. 2022Nonparametric Decoding for Generative Retrievalcited by 6
  13. 2022Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Modelscited by 6
  14. 2022Data-Efficient Autoregressive Document Retrieval for Fact Verificationcited by 6
  15. 2022Artefact Retrieval: Overview of NLP Models with Knowledge Base Accesscited by 4
  16. 2025RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasetscited by 4
  17. 2021Slot Filling for Biomedical Information Extractioncited by 3
  18. 2022KGI: An Integrated Framework for Knowledge Intensive Language Taskscited by 3
  19. 2022PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstandingcited by 3
  20. 2023INO at Factify 2: Structure Coherence based Multi-Modal Fact Verificationcited by 3
  21. 2022Fast and Light-Weight Answer Text Retrieval in Dialogue Systemscited by 2
  22. 2022Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrievalcited by 2
  23. 2025Multi-task retriever fine-tuning for domain-specific and efficient RAGcited by 2
  24. 2021Zero-shot Slot Filling with DPR and RAGcited by 1
  25. 2023Improving Multitask Retrieval by Promoting Task Specializationcited by 1
  26. 2023Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoderscited by 1
  27. 2025Deep Retrieval at CheckThat! 2025: Identifying Scientific Papers from Implicit Social Media Mentions via Hybrid Retrieval and Re-Rankingcited by 1
  28. 2021Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecturecited by 0
  29. 2022Fine-grained Czech News Article Dataset: An Interdisciplinary Approach to Trustworthiness Analysiscited by 0
  30. 2023Efficient Title Reranker for Fast and Improved Knowledge-Intense NLPcited by 0
  31. 2024Domain Adaptation of Multilingual Semantic Search -- Literature Reviewcited by 0
  32. 2025Application and Optimization of Large Models Based on Prompt Tuning for Fact-Check-Worthiness Estimationcited by 0