mlsanityData to 7 Oct 2026

Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

arXiv 2305.17306 ↗First posted 26 May 2023cs.CL cs.AI cs.LGcited by 102

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

As large language models (LLMs) are continuously being developed, their evaluation becomes increasingly important yet challenging. This work proposes Chain-of-Thought Hub, an open-source evaluation suite on the multi-step reasoning capabilities of large language models. We are interested in this setting for two reasons: (1) from the behavior of GPT and PaLM model family, we observe that complex reasoning is likely to be a key differentiator between weaker and stronger LLMs; (2) we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications, this naturally requires the foundation models to perform complex tasks that often involve the composition of linguistic and logical operations. Our approach is to compile a suite of challenging reasoning benchmarks to track the progress of LLMs. Our current results show that: (1) model scale clearly correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and PaLM-2 are the only two models that are comparable with GPT-4, while open-sourced models still lag behind; (3) LLaMA-65B performs closely to code-davinci-002, indicating that with successful further development such as reinforcement learning from human feedback (RLHF), it has great potential to be close to GPT-3.5-Turbo. Our results also suggest that for the open-source efforts to catch up, the community may focus more on building better base models and exploring RLHF.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 21 of its 35 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

21 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023PaLM 2 Technical Report17 May
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
7 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  7. 2023The False Promise of Imitating Proprietary LLMscited by 254

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 38; 2024: 42; 2025: 14; 2026: 8.

102 papers.

Sort:

By area:

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  4. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  5. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  8. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  9. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  10. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  11. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  12. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
38 more
  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  2. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  3. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 108
  4. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  5. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  6. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  7. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  8. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  9. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  10. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  11. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  12. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  13. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  14. 2023Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaborationcited by 48
  15. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  16. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correctioncited by 45
  17. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  18. 2025Rethinking Reflection in Pre-Trainingcited by 43
  19. 2024FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Modelscited by 36
  20. 2023Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Modelcited by 29
  21. 2023An In-depth Look at Gemini's Language Abilitiescited by 28
  22. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 28
  23. 2024Cycles of Thought: Measuring LLM Confidence through Stable Explanationscited by 27
  24. 2024LongGenBench: Long-context Generation Benchmarkcited by 27
  25. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  26. 2024Are Large Language Models Good Statisticians?cited by 26
  27. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Gamescited by 24
  28. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  29. 2023How susceptible are LLMs to Logical Fallacies?cited by 23
  30. 2023Mini-DALLE3: Interactive Text to Image by Prompting Large Language Modelscited by 22
  31. 2023WatME: Towards Lossless Watermarking Through Lexical Redundancycited by 22
  32. 2024MuMath-Code: Combining Tool-Use Large Language Models with Multi-perspective Data Augmentation for Mathematical Reasoningcited by 22
  33. 2023Critique Ability of Large Language Modelscited by 20
  34. 2024An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMscited by 20
  35. 2023When Do Program-of-Thoughts Work for Reasoning?cited by 19
  36. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  37. 2024Should We Really Edit Language Models? On the Evaluation of Edited Language Modelscited by 18
  38. 2023Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Modelscited by 17

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2024Are Your LLMs Capable of Stable Reasoning?
  4. 2026A Comprehensive Evaluation of LLM Reasoning: From Single-Model to Multi-Agent Paradigms
  5. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
  6. 2025Chain or tree? Re-evaluating complex reasoning from the perspective of a matrix of thought
  7. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  8. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning