mlsanityData to 7 Oct 2026

Boosting Language Models Reasoning with Chain-of-Knowledge Prompting

arXiv 2306.06427 ↗First posted 10 Jun 2023cs.CLcited by 77

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recently, Chain-of-Thought (CoT) prompting has delivered success on complex reasoning tasks, which aims at designing a simple prompt like ``Let's think step by step'' or multiple in-context exemplars with well-designed rationales to elicit Large Language Models (LLMs) to generate intermediate reasoning steps. However, the generated rationales often come with mistakes, making unfactual and unfaithful reasoning chains. To mitigate this brittleness, we propose a novel Chain-of-Knowledge (CoK) prompting, where we aim at eliciting LLMs to generate explicit pieces of knowledge evidence in the form of structure triple. This is inspired by our human behaviors, i.e., we can draw a mind map or knowledge map as the reasoning evidence in the brain before answering a complex question. Benefiting from CoK, we additionally introduce a F^2-Verification method to estimate the reliability of the reasoning chains in terms of factuality and faithfulness. For the unreliable response, the wrong evidence can be indicated to prompt the LLM to rethink. Extensive experiments demonstrate that our method can further improve the performance of commonsense, factual, symbolic, and arithmetic reasoning tasks.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 66 of its 117 references that are in the map. First posted on arXiv 10 Jun 2023.

Builds on

66 of 117 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Faithful Chain-of-Thought Reasoning31 Jan
  2. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters20 Dec
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  4. 2022Automatic Chain of Thought Prompting in Large Language Models7 Oct
  5. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  6. 2022Large Language Models are Zero-Shot Reasoners24 May
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  13. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  14. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  15. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  16. 2022Large Language Models Can Self-Improvecited by 656
  17. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  18. 2022PAL: Program-aided Language Modelscited by 635
  19. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  20. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  21. 2021MetaICL: Learning to Learn In Contextcited by 450
  22. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  23. 2016Solving General Arithmetic Word Problemscited by 353
  24. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  25. 2022Prompting GPT-3 To Be Reliablecited by 258
  26. 2023Do Large Language Models Know What They Don't Know?cited by 238
  27. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  28. 2021Internet-Augmented Dialogue Generationcited by 217
  29. 2020How Context Affects Language Models' Factual Predictionscited by 181
  30. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  31. 2021Meta-learning via Language Model In-context Tuningcited by 163
  32. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  33. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  34. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  35. 2023Compositional Exemplars for In-context Learningcited by 143
  36. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  37. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  38. 2019ASER: A Large-scale Eventuality Knowledge Graphcited by 89

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 26; 2025: 25; 2026: 11.

77 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  2. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  3. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  4. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  5. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  6. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 100
  7. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answeringcited by 97
  8. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  9. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  10. 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 77
  11. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  12. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
38 more
  1. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  2. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  3. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  4. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphscited by 32
  5. 2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 30
  6. 2025CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehensioncited by 23
  7. 2023Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinkingcited by 19
  8. 2024GraphOTTER: Evolving LLM-based Graph Reasoning for Complex Table Question Answeringcited by 18
  9. 2024An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaborationcited by 16
  10. 2025Large Language Models for Computer-Aided Design: A Surveycited by 16
  11. 2025KiRAG: Knowledge-Driven Iterative Retriever for Enhancing Retrieval-Augmented Generationcited by 12
  12. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  13. 2025Self-supervised Quantized Representation for Seamlessly Integrating Knowledge Graphs with Large Language Modelscited by 11
  14. 2024Graph Neural Network Enhanced Retrieval for Question Answering of LLMscited by 10
  15. 2023Semi-Structured Chain-of-Thought: Integrating Multiple Sources of Knowledge for Improved Language Model Reasoningcited by 9
  16. 2023Towards A Unified View of Answer Calibration for Multi-Step Reasoningcited by 8
  17. 2024ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphscited by 8
  18. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7
  19. 2024Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoningcited by 7
  20. 2024An Empirical Categorization of Prompting Techniques for Large Language Models: A Practitioner's Guidecited by 7
  21. 2024Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Modelscited by 7
  22. 2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learningcited by 7
  23. 2023Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learningcited by 6
  24. 2024Structure-aware Domain Knowledge Injection for Large Language Modelscited by 6
  25. 2024Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Pathscited by 6
  26. 2025Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendmentcited by 6
  27. 2025MIXRAG : Mixture-of-Experts Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 6
  28. 2024Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answeringcited by 5
  29. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Modelscited by 5
  30. 2024Towards Trustworthy Knowledge Graph Reasoning: An Uncertainty Aware Perspectivecited by 5
  31. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  32. 2024A Semantic Mention Graph Augmented Model for Document-Level Event Argument Extractioncited by 4
  33. 2024Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Modelscited by 4
  34. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  35. 2025How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptationcited by 4
  36. 2024LINKED: Eliciting, Filtering and Integrating Knowledge in Large Language Model for Commonsense Reasoningcited by 3
  37. 2025CDW-CoT: Clustered Distance-Weighted Chain-of-Thoughts Reasoningcited by 3
  38. 2025GraPPI: A Retrieve-Divide-Solve GraphRAG Framework for Large-scale Protein-protein Interaction Explorationcited by 3

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
  2. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  3. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  4. 2024Chain-of-Thought Reasoning Without Prompting
  5. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?
  6. 2026Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiency
  7. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods
  8. 2023Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models