mlsanityData to 7 Oct 2026

MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge

arXiv 2412.17032 ↗First posted 22 Dec 2024cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks but face significant challenges with complex, knowledge-intensive multi-hop queries, particularly those involving new or long-tail knowledge. Existing benchmarks often fail to fully address these challenges. To bridge this gap, we introduce MINTQA (Multi-hop Question Answering on New and Tail Knowledge), a comprehensive benchmark to evaluate LLMs' capabilities in multi-hop reasoning across four critical dimensions: question handling strategy, sub-question generation, retrieval-augmented generation, and iterative or dynamic decomposition and retrieval. MINTQA comprises 10,479 question-answer pairs for evaluating new knowledge and 17,887 pairs for assessing long-tail knowledge, with each question equipped with corresponding sub-questions and answers. Our systematic evaluation of 22 state-of-the-art LLMs on MINTQA reveals significant limitations in their ability to handle complex knowledge base queries, particularly in handling new or unpopular knowledge. Our findings highlight critical challenges and offer insights for advancing multi-hop reasoning capabilities. The MINTQA benchmark is available at https://github.com/probe2/multi-hop/.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

39 of 59 references.

References · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  11. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  12. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
27 more
  1. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  2. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  3. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  4. 2022Large Language Models with Controllable Working Memorycited by 169
  5. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  6. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 160
  7. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  8. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
  9. 2020A Survey of Knowledge-Enhanced Text Generationcited by 119
  10. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87
  11. 2022DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Basescited by 83
  12. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  13. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 71
  14. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  15. 2024Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answeringcited by 51
  16. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 49
  17. 2024RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedbackcited by 46
  18. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  19. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 39
  20. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  21. 2023Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoningcited by 21
  22. 2024Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answeringcited by 20
  23. 2024Deliberate Reasoning in Language Models as Structure-Aware Planning with an Accurate World Modelcited by 20
  24. 2024Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Modelscited by 19
  25. 2022Locate Then Ask: Interpretable Stepwise Reasoning for Multi-hop Question Answeringcited by 6
  26. 2019On the Possibilities and Limitations of Multi-hop Reasoning Under Linguistic Imperfectionscited by 3
  27. 2020Learning to Recover Reasoning Chains for Multi-Hop Question Answering via Cooperative Gamescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  2. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  3. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  4. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  5. 2025Agent models: Internalizing Chain-of-Action Generation into Reasoning modelscited by 17
  6. 2025InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generationcited by 15
  7. 2025M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmarkcited by 11
  8. 2026CREATE: Testing LLMs for Associative Creativitycited by 6
  9. 2026AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAGcited by 5
  10. 2025KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integrationcited by 4
  11. 2025Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metricscited by 4
  12. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answeringcited by 3
8 more
  1. 2025Beyond GeneGPT: A Multi-Agent Architecture with Open-Source LLMs for Enhanced Genomic Question Answeringcited by 3
  2. 2025HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questionscited by 2
  3. 2025Fine-Grained Evaluation for Implicit Discourse Relation Recognitioncited by 1
  4. 2025CompoST: A Benchmark for Analyzing the Ability of LLMs To Compositionally Interpret Questions in a QALD Settingcited by 1
  5. 2025MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inferencecited by 1
  6. 2026ARMove: Learning to Predict Human Mobility through Agentic Reasoningcited by 1
  7. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  8. 2026Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models
  2. 2026MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA
  3. 2024FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models
  4. 2026Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge
  5. 2025StepChain GraphRAG: Reasoning Over Knowledge Graphs for Multi-Hop Question Answering
  6. 2023PokeMQA: Programmable knowledge editing for Multi-hop Question Answering
  7. 2025From Query to Logic: Ontology-Driven Multi-Hop Reasoning in LLMs
  8. 2025iQUEST: An Iterative Question-Guided Framework for Knowledge Base Question Answering