# MetaQA: Combining Expert Agents for Multi-Skill Question Answering

arXiv: https://arxiv.org/abs/2112.01922 · HTML: https://mlsanity.com/paper/2112.01922/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension](https://mlsanity.com/paper/1804.07927/) (2018)
- [MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension](https://mlsanity.com/paper/1905.13453/) (2019)

## Built on by (most cited first, 15 of 15)

- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering](https://mlsanity.com/paper/2410.08085/) (2024)
- [A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications](https://mlsanity.com/paper/2510.16724/) (2025)
- [SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA](https://mlsanity.com/paper/2409.16682/) (2024)
- [UKP-SQUARE: An Online Platform for Question Answering Research](https://mlsanity.com/paper/2203.13693/) (2022)
- [Multi-LLM QA with Embodied Exploration](https://mlsanity.com/paper/2406.10918/) (2024)
- [CLAUSE: Agentic Neuro-Symbolic Knowledge Graph Reasoning via Dynamic Learnable Context Engineering](https://mlsanity.com/paper/2509.21035/) (2025)
- [UKP-SQuARE v2: Explainability and Adversarial Attacks for Trustworthy QA](https://mlsanity.com/paper/2208.09316/) (2022)
- [Schema-Guided Culture-Aware Complex Event Simulation with Multi-Agent Role-Play](https://mlsanity.com/paper/2410.18935/) (2024)
- [UKP-SQuARE v3: A Platform for Multi-Agent QA Research](https://mlsanity.com/paper/2303.18120/) (2023)
- [HERA: Hybrid Edge-cloud Resource Allocation for Cost-Efficient AI Agents](https://mlsanity.com/paper/2504.00434/) (2025)
- [KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge](https://mlsanity.com/paper/2508.14080/) (2025)
- [RELOOP: Recursive Retrieval with Multi-Hop Reasoner and Planners for Heterogeneous QA](https://mlsanity.com/paper/2510.20505/) (2025)
- [UKP-SQuARE: An Interactive Tool for Teaching Question Answering](https://mlsanity.com/paper/2305.19748/) (2023)

## Abstract (arXiv)

The recent explosion of question answering (QA) datasets and models has increased the interest in the generalization of models across multiple domains and formats by either training on multiple datasets or by combining multiple models. Despite the promising results of multi-dataset models, some domains or QA formats may require specific architectures, and thus the adaptability of these models might be limited. In addition, current approaches for combining models disregard cues such as question-answer compatibility. In this work, we propose to combine expert agents with a novel, flexible, and training-efficient architecture that considers questions, answer predictions, and answer-prediction confidence scores to select the best answer among a list of answer candidates. Through quantitative and qualitative experiments we show that our model i) creates a collaboration between agents that outperforms previous multi-agent and multi-dataset approaches in both in-domain and out-of-domain scenarios, ii) is highly data-efficient to train, and iii) can be adapted to any QA format. We release our code and a dataset of answer predictions from expert agents for 16 QA datasets to foster future developments of multi-agent systems on https://github.com/UKPLab/MetaQA.