Improving Multitask Retrieval by Promoting Task Specialization
Not on the map. Why?
Abstract (arXiv)
In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval in which a separate retriever is trained for each task. We show that it is possible to train a multitask retriever that outperforms task-specific retrievers by promoting task specialization. The main ingredients are: (1) a better choice of pretrained model (one that is explicitly optimized for multitasking) along with compatible prompting, and (2) a novel adaptive learning method that encourages each parameter to specialize in a particular task. The resulting multitask retriever is highly performant on the KILT benchmark. Upon analysis, we find that the model indeed learns parameters that are more task-specialized compared to naive multitasking without prompting or adaptive learning.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 1 Jul 2023.
Builds on
27 of 36 references.
References · 27
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
- 2019Importance Estimation for Neural Network Pruningcited by 531
15 more
- 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
- 2020Autoregressive Entity Retrievalcited by 333
- 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
- 2019Zero-Shot Entity Linking by Reading Entity Descriptionscited by 124
- 2019Learning Dense Representations for Entity Retrievalcited by 122
- 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
- 2022Task-aware Retrieval with Instructionscited by 107
- 2019Scalable Zero-shot Entity Linking with Dense Entity Retrievalcited by 85
- 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
- 2021Multi-task Retrieval for Knowledge-Intensive Taskscited by 44
- 2022CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Taskscited by 36
- 2021Focus on the Common Good: Group Distributional Robustness Followscited by 25
- 2022No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Modelscited by 15
- 2022TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrievalcited by 8
- 2021OpenMatch: An Open Source Library for Neu-IR Researchcited by 6
Built on by
Citing papers per year: 2024: 1.
1 paper.