# CompMix: A Benchmark for Heterogeneous Question Answering

arXiv: https://arxiv.org/abs/2306.12235 · HTML: https://mlsanity.com/paper/2306.12235/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2023.

## Built on by (most cited first, 12 of 12)

- [Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge](https://mlsanity.com/paper/2403.01432/) (2024)
- [SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing](https://mlsanity.com/paper/2406.00562/) (2024)
- [RAG-based Question Answering over Heterogeneous Data and Text](https://mlsanity.com/paper/2412.07420/) (2024)
- [DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text](https://mlsanity.com/paper/2310.20170/) (2023)
- [UniHGKR: Unified Instruction-aware Heterogeneous Knowledge Retrievers](https://mlsanity.com/paper/2410.20163/) (2024)
- [mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs](https://mlsanity.com/paper/2505.11180/) (2025)
- [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025)
- [DBRouting: Routing End User Queries to Databases for Answerability](https://mlsanity.com/paper/2501.16220/) (2025)
- [ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models](https://mlsanity.com/paper/2408.05948/) (2024)
- [ER-RAG: Enhance RAG with ER-Based Unified Modeling of Heterogeneous Data Sources](https://mlsanity.com/paper/2504.06271/) (2025)
- [Detecting Knowledge Inconsistencies Across Text, Tables, and Knowledge Graphs](https://mlsanity.com/paper/2607.25959/) (2026)
- [Benchmarking Hybrid Deep Research Across Database Querying and Web Search](https://mlsanity.com/paper/2609.09410/) (2026)

## Abstract (arXiv)

Fact-centric question answering (QA) often requires access to multiple, heterogeneous, information sources. By jointly considering several sources like a knowledge base (KB), a text collection, and tables from the web, QA systems can enhance their answer coverage and confidence. However, existing QA benchmarks are mostly constructed with a single source of knowledge in mind. This limits capabilities of these benchmarks to fairly evaluate QA systems that can tap into more than one information repository. To bridge this gap, we release CompMix, a crowdsourced QA benchmark which naturally demands the integration of a mixture of input sources. CompMix has a total of 9,410 questions, and features several complex intents like joins and temporal conditions. Evaluation of a range of QA systems on CompMix highlights the need for further research on leveraging information from heterogeneous sources.