# MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale

arXiv: https://arxiv.org/abs/2010.00980 · HTML: https://mlsanity.com/paper/2010.00980/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [xGQA: Cross-Lingual Visual Question Answering](https://mlsanity.com/paper/2109.06082/) (2021)
- [What to Pre-Train on? Efficient Intermediate Task Selection](https://mlsanity.com/paper/2104.08247/) (2021)
- [PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains](https://mlsanity.com/paper/2102.12206/) (2021)
- [TWEAC: Transformer with Extendable QA Agent Classifiers](https://mlsanity.com/paper/2104.07081/) (2021)
- [mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations](https://mlsanity.com/paper/2305.14224/) (2023)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Active Learning Over Multiple Domains in Natural Language Tasks](https://mlsanity.com/paper/2202.00254/) (2022)
- [Divide and Conquer: Text Semantic Matching with Disentangled Keywords and Intents](https://mlsanity.com/paper/2203.02898/) (2022)
- [A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models](https://mlsanity.com/paper/2103.03335/) (2021)
- [AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters](https://mlsanity.com/paper/2108.08103/) (2021)
- [Modularized Transfer Learning with Multiple Knowledge Graphs for Zero-shot Commonsense Reasoning](https://mlsanity.com/paper/2206.03715/) (2022)
- [Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets](https://mlsanity.com/paper/2205.11472/) (2022)
- [Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters](https://mlsanity.com/paper/2212.10448/) (2022)
- [Improving Text Semantic Similarity Modeling through a 3D Siamese Network](https://mlsanity.com/paper/2307.09274/) (2023)
- [Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision](https://mlsanity.com/paper/2107.08582/) (2021)
- [DREditor: An Time-efficient Approach for Building a Domain-specific Dense Retrieval Model](https://mlsanity.com/paper/2401.12540/) (2024)
- [Harnessing Multilingual Resources to Question Answering in Arabic](https://mlsanity.com/paper/2205.08024/) (2022)
- [Massively Multilingual Lexical Specialization of Multilingual Transformers](https://mlsanity.com/paper/2208.01018/) (2022)

## Abstract (arXiv)

We study the zero-shot transfer capabilities of text matching models on a massive scale, by self-supervised training on 140 source domains from community question answering forums in English. We investigate the model performances on nine benchmarks of answer selection and question similarity tasks, and show that all 140 models transfer surprisingly well, where the large majority of models substantially outperforms common IR baselines. We also demonstrate that considering a broad selection of source domains is crucial for obtaining the best zero-shot transfer performances, which contrasts the standard procedure that merely relies on the largest and most similar domains. In addition, we extensively study how to best combine multiple source domains. We propose to incorporate self-supervised with supervised multi-task learning on all available source domains. Our best zero-shot transfer model considerably outperforms in-domain BERT and the previous state of the art on six benchmarks. Fine-tuning of our model with in-domain data results in additional large gains and achieves the new state of the art on all nine benchmarks.