# MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge

arXiv: https://arxiv.org/abs/2412.17032 · HTML: https://mlsanity.com/paper/2412.17032/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 59 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Builds on (most cited first, 30 of 39)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval](https://mlsanity.com/paper/2009.12756/) (2020)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)
- [Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework](https://mlsanity.com/paper/2305.03268/) (2023)
- [A Survey of Knowledge-Enhanced Text Generation](https://mlsanity.com/paper/2010.04389/) (2020)
- [Compositional Questions Do Not Necessitate Multi-hop Reasoning](https://mlsanity.com/paper/1906.02900/) (2019)
- [DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases](https://mlsanity.com/paper/2210.00063/) (2022)
- [Searching for Best Practices in Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01219/) (2024)
- [Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering](https://mlsanity.com/paper/2210.01613/) (2022)
- [When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation](https://mlsanity.com/paper/2402.11457/) (2024)
- [Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answering](https://mlsanity.com/paper/2406.14891/) (2024)
- [Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge](https://mlsanity.com/paper/2403.01432/) (2024)
- [RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback](https://mlsanity.com/paper/2403.06840/) (2024)
- [LM-Cocktail: Resilient Tuning of Language Models via Model Merging](https://mlsanity.com/paper/2311.13534/) (2023)

## Built on by (most cited first, 20 of 20)

- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [SMART: Self-Aware Agent for Tool Overuse Mitigation](https://mlsanity.com/paper/2502.11435/) (2025)
- [Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs](https://mlsanity.com/paper/2507.09477/) (2025)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [Agent models: Internalizing Chain-of-Action Generation into Reasoning models](https://mlsanity.com/paper/2503.06580/) (2025)
- [InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.15872/) (2025)
- [M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark](https://mlsanity.com/paper/2511.17729/) (2025)
- [CREATE: Testing LLMs for Associative Creativity](https://mlsanity.com/paper/2603.09970/) (2026)
- [AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG](https://mlsanity.com/paper/2602.19127/) (2026)
- [KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration](https://mlsanity.com/paper/2508.15790/) (2025)
- [Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics](https://mlsanity.com/paper/2510.05137/) (2025)
- [Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering](https://mlsanity.com/paper/2505.16591/) (2025)
- [Beyond GeneGPT: A Multi-Agent Architecture with Open-Source LLMs for Enhanced Genomic Question Answering](https://mlsanity.com/paper/2511.15061/) (2025)
- [HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions](https://mlsanity.com/paper/2505.15087/) (2025)
- [Fine-Grained Evaluation for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2503.05326/) (2025)
- [CompoST: A Benchmark for Analyzing the Ability of LLMs To Compositionally Interpret Questions in a QALD Setting](https://mlsanity.com/paper/2507.21257/) (2025)
- [MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference](https://mlsanity.com/paper/2509.22750/) (2025)
- [ARMove: Learning to Predict Human Mobility through Agentic Reasoning](https://mlsanity.com/paper/2604.17419/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain](https://mlsanity.com/paper/2607.20510/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks but face significant challenges with complex, knowledge-intensive multi-hop queries, particularly those involving new or long-tail knowledge. Existing benchmarks often fail to fully address these challenges. To bridge this gap, we introduce MINTQA (Multi-hop Question Answering on New and Tail Knowledge), a comprehensive benchmark to evaluate LLMs' capabilities in multi-hop reasoning across four critical dimensions: question handling strategy, sub-question generation, retrieval-augmented generation, and iterative or dynamic decomposition and retrieval. MINTQA comprises 10,479 question-answer pairs for evaluating new knowledge and 17,887 pairs for assessing long-tail knowledge, with each question equipped with corresponding sub-questions and answers. Our systematic evaluation of 22 state-of-the-art LLMs on MINTQA reveals significant limitations in their ability to handle complex knowledge base queries, particularly in handling new or unpopular knowledge. Our findings highlight critical challenges and offer insights for advancing multi-hop reasoning capabilities. The MINTQA benchmark is available at https://github.com/probe2/multi-hop/.