mlsanityData to 7 Oct 2026

Reasoning benchmarks and evaluation

2,388 papers3 this weekLow confidence in this grouping: the papers may belong to more than one topic, or the group may changeLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About

Keywords: language, reasoning, large language, large, llms

Key facts. Reasoning benchmarks and evaluation: 2,388 papers on arXiv since 2021 (data through 7 Oct 2026). 230 of its 2,388 mapped papers (10%) cite “Qwen3 Technical Report” (2025, arXiv 2505.09388). 177 of its 2,388 mapped papers (7%) cite “LLaMA: Open and Efficient Foundation Language Models” (2023, arXiv 2302.13971). 41 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

41 new papers, 17 Sep – 7 Oct; 3 this week.

  1. This week · 3
  2. The Long Road to the Same Answer: Cognitive Bias Under Escalating Reasoning Budgets in Large Language Models also fits: Prompting and chain-of-thought reasoning
  3. Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs also fits: Prompting and chain-of-thought reasoning
  4. Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspective
  5. Earlier · 38
  6. Verb-ICL: Rethinking In-Context Learning for Structured Prediction also fits: Test-time compute and efficient thinking
  7. Language-Conditioned Token and Reasoning Efficiency in Large Language Models: A Paired Cross-Lingual Study Protocol also fits: Test-time compute and efficient thinking
  8. Writerslogic at PAN 2026: Process over Content for Robust Detection under Domain Shift also fits: Reasoning for agents, planning and code
  9. SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models
  10. Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation also fits: Knowledge- and tool-augmented reasoning
  11. Evaluating Multi-Dimensional Generalization of Large Language Models in Temporal Extraction Tasks
  12. Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning also fits: Reasoning for agents, planning and code
All 41
  1. The Concrete-Arbitrary Gap: Kinship Reasoning in LLMs Is Not Indifferent to Presentation
  2. Cognitive Enhancement: Rethinking the Necessity of Role-Playing for Large Language Models
  3. Also Small Models Can Reasonably Self-Evaluate Their Confidence
  4. Offline Guidance, Online Reasoning: Reusing LLM Feedback for Small Language Models also fits: Test-time compute and efficient thinking
  5. When Order Matters: First-Speaker Bias and Mitigation through Personality in Sequential Multi-Agent Debate also fits: Reasoning for agents, planning and code
  6. ACTR: Aligning Thoughts and Responses for Multilingual Safety in Reasoning LLMs also fits: Reasoning for agents, planning and code
  7. LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning also fits: Knowledge- and tool-augmented reasoning
  8. Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs also fits: Test-time compute and efficient thinking
  9. Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
  10. What Does Post-Training Change in Multilingual Reasoning?
  11. Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation also fits: Reasoning for agents, planning and code
  12. Layer-Informed Fine-Tuning via Three-Stage Functional Segmentation of LLMs also fits: Test-time compute and efficient thinking
  13. Mnemon: Raw Records, Fast Judgments, Slow Thoughts also fits: Test-time compute and efficient thinking
  14. MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs
  15. MechReasoner: A Simulator and Benchmark for Mechanistic Reasoning in Qualitative Physics also fits: Test-time compute and efficient thinking
  16. When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation also fits: Test-time compute and efficient thinking
  17. Quizzing the Translation: A Prover-Grounded Evaluation Metric for NLrightarrowFOL also fits: Mathematical reasoning
  18. Certainty Is Not Just Correctness: Rethinking Token-Level Certainty in LLM Reasoning also fits: Test-time compute and efficient thinking
  19. Counting on Thinking: Tracing Evidence Integration in Language Models also fits: Prompting and chain-of-thought reasoning
  20. Locally Sound, Globally Insufficient: The Local-Global Gap in Multi-Hop Reasoning also fits: Test-time compute and efficient thinking
  21. Reasoning Concentrates Errors, and Self-Consistency Never Notices also fits: Test-time compute and efficient thinking
  22. From annotation to reasoning: Culture in language models
  23. Enhancing Assessment of Self-Consistency in LLM Explanations using Perturbation Strength also fits: Test-time compute and efficient thinking
  24. Route-Verify-Vote: Procedure-Conditioned Self-Consistency for Mixed-Domain Reasoning also fits: Test-time compute and efficient thinking
  25. Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models also fits: Knowledge- and tool-augmented reasoning
  26. Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting
  27. GRACE: Grounded Adversarial Reasoning over Canadian Law also fits: Reasoning for agents, planning and code
  28. LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator also fits: Test-time compute and efficient thinking
  29. The Communication Bottleneck: A Round-Trip Study of Tree-Structured Expression Serialization in Language Models also fits: Prompting and chain-of-thought reasoning
  30. When Does Reasoning Help in Machine Translation? A Hierarchical Analysis of LRM Reasoning Traces
  31. Language-model groups overstate consensus when replaying human deliberation on a reasoning task

Foundations

Most cited by papers here

  1. Qwen3 Technical Report2025cited by 230 here
  2. LLaMA: Open and Efficient Foundation Language Models2023cited by 177 here
  3. FOLIO: Natural Language Reasoning with First-Order Logic2022cited by 139 here
  4. ReAct: Synergizing Reasoning and Acting in Language Models2022cited by 139 here
  5. Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning2023cited by 126 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 857 here
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 354 here
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 366 here
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 139 here
  5. 2022PAL: Program-aided Language Modelscited by 95 here
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 142 here
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 281 here
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 183 here
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 382 here
15 more it builds on, not in this chain
  1. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 147 here
  2. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 139 here
  3. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 123 here
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 122 here
  5. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 112 here
  6. 2022Towards Reasoning in Large Language Models: A Surveycited by 105 here
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 177 here
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 118 here
  9. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 126 here
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 112 here
  11. 2023Let's Verify Step by Stepcited by 140 here
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 103 here
  13. 2024OpenAI o1 System Cardcited by 114 here
  14. 2024DeepSeek-V3 Technical Reportcited by 114 here
  15. 2025Qwen3 Technical Reportcited by 230 here

The 24 most cited of 3,150.

Browse

30 of 2,388, ranked 6–35

  1. 6Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought2022123
  2. 7DeepSeek-V3 Technical Report2024114
  3. 8Towards Reasoning in Large Language Models: A Survey2022105
  4. 9PAL: Program-aided Language Models202295
  5. 10Show Your Work: Scratchpads for Intermediate Computation with Language Models202192
  6. 11LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers202379
  7. 12Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning202269
  8. 13Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks202355
  9. 14Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?202248
  10. 15LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models202445
All 30
  1. 16Emergent Analogical Reasoning in Large Language Models202245
  2. 17Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting202344
  3. 18Language models show human-like content effects on reasoning tasks202244
  4. 19Improving Factuality and Reasoning in Language Models through Multiagent Debate202341
  5. 20Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples202341
  6. 21A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity202339
  7. 22On the Paradox of Learning to Reason from Data202239
  8. 23Measuring and Narrowing the Compositionality Gap in Language Models202239
  9. 24SatLM: Satisfiability-Aided Language Models Using Declarative Prompting202336
  10. 25Understanding Social Reasoning in Language Models with Language Models202336
  11. 26Dissociating language and thought in large language models202334
  12. 27PaLM 2 Technical Report202333
  13. 28ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning202233
  14. 29Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models202531
  15. 30ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning202531
  16. 31MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization202431
  17. 32Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4202331
  18. 33Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond202331
  19. 34Code Llama: Open Foundation Models for Code202331
  20. 35LAMBADA: Backward Chaining for Automated Reasoning in Natural Language202231