mlsanityData to 7 Oct 2026

MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning

arXiv 2310.16049 ↗First posted 24 Oct 2023cs.CLcited by 195

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

While large language models (LLMs) equipped with techniques like chain-of-thought prompting have demonstrated impressive capabilities, they still fall short in their ability to reason robustly in complex settings. However, evaluating LLM reasoning is challenging because system capabilities continue to grow while benchmark datasets for tasks like logical deduction have remained static. We introduce MuSR, a dataset for evaluating language models on multistep soft reasoning tasks specified in a natural language narrative. This dataset has two crucial features. First, it is created through a novel neurosymbolic synthetic-to-natural generation algorithm, enabling the construction of complex reasoning instances that challenge GPT-4 (e.g., murder mysteries roughly 1000 words in length) and which can be scaled further as more capable LLMs are released. Second, our dataset instances are free text narratives corresponding to real-world domains of reasoning; this makes it simultaneously much more challenging than other synthetically-crafted benchmarks while remaining realistic and tractable for human annotators to solve with high accuracy. We evaluate a range of LLMs and prompting techniques on this dataset and characterize the gaps that remain for techniques like chain-of-thought to perform robust reasoning.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 7 of its 47 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

7 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Understanding Social Reasoning in Language Models with Language Models21 Jun
  2. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information13 Jun
  3. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples24 May
  4. 2022PAL: Program-aided Language Models18 Nov

Other references · 3

  1. 2023How Language Model Hallucinations Can Snowballcited by 310
  2. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  3. 2023Faithful Chain-of-Thought Reasoningcited by 285

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 86; 2026: 84.

195 papers.

Sort:

By area:

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  2. 2025Why Language Models Hallucinatecited by 205
  3. 2025Survey on Evaluation of LLM-based Agentscited by 190
  4. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  5. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  6. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  7. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  8. 2025Voting or Consensus? Decision-Making in Multi-Agent Debatecited by 55
  9. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  10. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  11. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  12. 2025A Survey of Direct Preference Optimizationcited by 35
38 more
  1. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?cited by 34
  2. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  3. 2025CARROT: A Cost Aware Rate Optimal Routercited by 34
  4. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
  5. 2024Hermes 3 Technical Reportcited by 33
  6. 2025Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lenscited by 32
  7. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  8. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Modelscited by 29
  9. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  10. 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 27
  11. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  12. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  13. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 23
  14. 2025JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Modelscited by 21
  15. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 21
  16. 2024LLM-Generated Black-box Explanations Can Be Adversarially Helpfulcited by 20
  17. 2025Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluationcited by 20
  18. 2024CodeUpdateArena: Benchmarking Knowledge Editing on API Updatescited by 18
  19. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  20. 2025SEA-LION: Southeast Asian Languages in One Networkcited by 17
  21. 2025Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLMcited by 16
  22. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 16
  23. 2025MALLM: Multi-Agent Large Language Models Frameworkcited by 15
  24. 2025PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluationcited by 14
  25. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  26. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  27. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  28. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  29. 2026Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillationcited by 13
  30. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 12
  31. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Modelscited by 12
  32. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 11
  33. 2025Privacy Reasoning in Ambiguous Contextscited by 11
  34. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  35. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  36. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 9
  37. 2025Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoningcited by 9
  38. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMscited by 9

50 most cited of 195 All 195 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2023ThoughtSource: A central hub for large language model reasoning data
  4. 2024Reliable Reasoning Beyond Natural Language
  5. 2025Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean
  6. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
  7. 2024P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains
  8. 2025Theorem-of-Thought: A Multi-Agent Framework for Abductive, Deductive, and Inductive Reasoning in Language Models