mlsanityData to 7 Oct 2026

Answering Questions by Meta-Reasoning over Multiple Chains of Thought

arXiv 2304.13007 ↗First posted 25 Apr 2023cs.CL cs.AIcited by 87

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Modern systems for multi-hop question answering (QA) typically break questions into a sequence of reasoning steps, termed chain-of-thought (CoT), before arriving at a final answer. Often, multiple chains are sampled and aggregated through a voting mechanism over the final answers, but the intermediate steps themselves are discarded. While such approaches improve performance, they do not consider the relations between intermediate steps across chains and do not provide a unified explanation for the predicted answer. We introduce Multi-Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought, rather than aggregating their answers. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in generating an explanation and predicting the answer. MCR outperforms strong baselines on 7 multi-hop QA datasets. Moreover, our analysis reveals that MCR explanations exhibit high quality, enabling humans to verify its answers.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 67 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

53 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  6. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  12. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  13. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  14. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  15. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  16. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  17. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  18. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  19. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  20. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  21. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  22. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  23. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  24. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  25. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  26. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  27. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  28. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  29. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 166
  30. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 147
  31. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  32. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  33. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  34. 2022Language Model Cascadescited by 104
  35. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  36. 2019QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questionscited by 89
  37. 2022Recitation-Augmented Language Modelscited by 65
  38. 2020IIRC: A Dataset of Incomplete Information Reading Comprehension Questionscited by 52

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 29; 2025: 21; 2026: 7.

87 papers.

Sort:

By area:

  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  2. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  3. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  4. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  5. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  6. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  7. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  8. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  9. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  10. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  11. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  12. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
38 more
  1. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  2. 2025Confidence Improves Self-Consistency in LLMscited by 111
  3. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  4. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  5. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  6. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  7. 2023Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Samplingcited by 46
  8. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Modelscited by 45
  9. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44
  10. 2023Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Modelscited by 37
  11. 2024RAR-b: Reasoning as Retrieval Benchmarkcited by 33
  12. 2025Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Modelscited by 33
  13. 2023The ART of LLM Refinement: Ask, Refine, and Trustcited by 29
  14. 2024BlendFilter: Advancing Retrieval-Augmented Large Language Models via Query Generation Blending and Knowledge Filteringcited by 29
  15. 2024STEER: Assessing the Economic Rationality of Large Language Modelscited by 26
  16. 2024I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Tokencited by 26
  17. 2023Flows: Building Blocks of Reasoning and Collaborating AIcited by 25
  18. 2023Diversity of Thought Improves Reasoning Abilities of LLMscited by 24
  19. 2023Mathematical Language Models: A Surveycited by 24
  20. 2024XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Modelscited by 23
  21. 2023Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoningcited by 22
  22. 2024Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatabilitycited by 22
  23. 2024Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generationcited by 21
  24. 2025Process-based Self-Rewarding Language Modelscited by 21
  25. 2023Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questionscited by 19
  26. 2024Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Modelscited by 17
  27. 2024LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generationcited by 17
  28. 2024Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMscited by 17
  29. 2024Mirror-Consistency: Harnessing Inconsistency in Majority Votingcited by 16
  30. 2024Can We Verify Step by Step for Incorrect Answer Detection?cited by 15
  31. 2024ERAGent: Enhancing Retrieval-Augmented Language Models with Improved Accuracy, Efficiency, and Personalizationcited by 15
  32. 2024Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systemscited by 15
  33. 2025EnrichIndex: Using LLMs to Enrich Retrieval Indices Offlinecited by 15
  34. 2024Hierarchical Retrieval-Augmented Generation Model with Rethink for Multi-hop Question Answeringcited by 14
  35. 2024Efficient Prompting for LLM-based Generative Internet of Thingscited by 12
  36. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMscited by 12
  37. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  38. 2023Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Modelscited by 11

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs
  2. 2026Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trends
  3. 2024Tree of Reviews: A Tree-based Dynamic Iterative Retrieval Framework for Multi-hop Question Answering
  4. 2025StepChain GraphRAG: Reasoning Over Knowledge Graphs for Multi-Hop Question Answering
  5. 2025BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answering
  6. 2024BeamAggR: Beam Aggregation Reasoning over Multi-source Knowledge for Multi-hop Question Answering
  7. 2025Semantic Bridge: Universal Multi-Hop Question Generation via AMR-Driven Graph Synthesis
  8. 2026BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence