# MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models

arXiv: https://arxiv.org/abs/2005.02507 · HTML: https://mlsanity.com/paper/2005.02507/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2020.

## Built on by (most cited first, 23 of 23)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [TWEAC: Transformer with Extendable QA Agent Classifiers](https://mlsanity.com/paper/2104.07081/) (2021)
- [Reasoning over Public and Private Data in Retrieval-Based Systems](https://mlsanity.com/paper/2203.11027/) (2022)
- [Exploring Dual Encoder Architectures for Question Answering](https://mlsanity.com/paper/2204.07120/) (2022)
- [An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability](https://mlsanity.com/paper/2109.05327/) (2021)
- [Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence](https://mlsanity.com/paper/2503.05037/) (2025)
- [SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives](https://mlsanity.com/paper/2306.02516/) (2023)
- [A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models](https://mlsanity.com/paper/2103.03335/) (2021)
- [Learning to Reuse Distractors to support Multiple Choice Question Generation in Education](https://mlsanity.com/paper/2210.13964/) (2022)
- [Policy-Gradient Training of Language Models for Ranking](https://mlsanity.com/paper/2310.04407/) (2023)
- [M2QA: Multi-domain Multilingual Question Answering](https://mlsanity.com/paper/2407.01091/) (2024)
- [DEXTER: A Benchmark for open-domain Complex Question Answering using LLMs](https://mlsanity.com/paper/2406.17158/) (2024)
- [Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges](https://mlsanity.com/paper/2211.14732/) (2022)
- [Mafin: Enhancing Black-Box Embeddings with Model Augmented Fine-Tuning](https://mlsanity.com/paper/2402.12177/) (2024)
- [Backtracing: Retrieving the Cause of the Query](https://mlsanity.com/paper/2403.03956/) (2024)
- [K-COMP: Retrieval-Augmented Medical Domain Question Answering With Knowledge-Injected Compressor](https://mlsanity.com/paper/2501.13567/) (2025)
- [Zero-Shot Estimation of Base Models' Weights in Ensemble of Machine Reading Comprehension Systems for Robust Generalization](https://mlsanity.com/paper/2106.16013/) (2021)
- [JCSE: Contrastive Learning of Japanese Sentence Embeddings and Its Applications](https://mlsanity.com/paper/2301.08193/) (2023)
- [Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generation](https://mlsanity.com/paper/2503.09094/) (2025)
- [Towards Consistency Filtering-Free Unsupervised Learning for Dense Retrieval](https://mlsanity.com/paper/2308.02926/) (2023)
- [Semantically Orthogonal Framework for Citation Classification: Disentangling Intent and Content](https://mlsanity.com/paper/2601.05103/) (2026)
- [Retrieval Sensitivity to Identity Signals in Queries](https://mlsanity.com/paper/2609.36534/) (2026)

## Abstract (arXiv)

Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al.,2019).This paper presents MultiReQA, anew multi-domain ReQA evaluation suite com-posed of eight retrieval QA tasks drawn from publicly available QA datasets. We provide the first systematic retrieval based evaluation over these datasets using two supervised neural models, based on fine-tuning BERT andUSE-QA models respectively, as well as a surprisingly strong information retrieval baseline,BM25. Five of these tasks contain both train-ing and test data, while three contain test data only. Performance on the five tasks with train-ing data shows that while a general model covering all domains is achievable, the best performance is often obtained by training exclusively on in-domain data.