mlsanityData to 7 Oct 2026

ThoughtSource: A central hub for large language model reasoning data

arXiv 2301.11596 ↗First posted 27 Jan 2023cs.CL cs.AIcited by 23

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large language models (LLMs) such as GPT-4 have recently demonstrated impressive results across a wide range of tasks. LLMs are still limited, however, in that they frequently fail at complex reasoning, their reasoning processes are opaque, they are prone to 'hallucinate' facts, and there are concerns about their underlying biases. Letting models verbalize reasoning steps as natural language, a technique known as chain-of-thought prompting, has recently been proposed as a way to address some of these issues. Here we present ThoughtSource, a meta-dataset and software library for chain-of-thought (CoT) reasoning. The goal of ThoughtSource is to improve future artificial intelligence systems by facilitating qualitative understanding of CoTs, enabling empirical evaluations, and providing training data. This first release of ThoughtSource integrates seven scientific/medical, three general-domain and five math word question answering datasets.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 43 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

34 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May
  2. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations24 May
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
18 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  4. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  6. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  11. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  12. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  13. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  14. 2022Can large language models reason about medical questions?cited by 194
  15. 2021Explaining Answers with Entailment Treescited by 148
  16. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  17. 2020QED: A Framework and Dataset for Explanations in Question Answeringcited by 55
  18. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 7; 2025: 8.

23 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  4. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 63
  5. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  6. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  7. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 28
  8. 2025DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Taskscited by 24
  9. 2023ChatGPT: More than a Weapon of Mass Deception, Ethical challenges and responses from the Human-Centered Artificial Intelligence (HCAI) perspectivecited by 22
  10. 2025A Theory of Learning with Autoregressive Chain of Thoughtcited by 20
  11. 2023GLoRE: Evaluating Logical Reasoning of Large Language Modelscited by 13
  12. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMscited by 12
11 more
  1. 2024FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoningcited by 12
  2. 2025Structured Agent Distillation for Large Language Modelcited by 11
  3. 2025No Need for Explanations: LLMs can implicitly learn from mistakes in-contextcited by 10
  4. 2025Proof-of-TBI -- Fine-Tuned Vision Language Model Consortium and OpenAI-o3 Reasoning LLM-Based Medical Diagnosis Support System for Mild Traumatic Brain Injury (TBI) Predictioncited by 10
  5. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Reviewcited by 8
  6. 2024Rationale-Aware Answer Verification by Pairwise Self-Evaluationcited by 8
  7. 2025Do LLM Agents Know How to Ground, Recover, and Assess? A Benchmark for Epistemic Competence in Information-Seeking Agentscited by 7
  8. 2023An automatically discovered chain-of-thought prompt generalizes to novel models and datasetscited by 5
  9. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Modelscited by 2
  10. 2024Rethinking Thinking Tokens: Understanding Why They Underperform in Practicecited by 2
  11. 2025Small Language Models Offer Significant Potential for Science Communitycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
  2. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  3. 2025Early Stopping Chain-of-thoughts in Large Language Models
  4. 2025Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotations
  5. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  6. 2024General Purpose Verification for Chain of Thought Prompting
  7. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Prompting
  8. 2025Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning