mlsanityData to 14 Oct 2026

Are Your LLMs Capable of Stable Reasoning?

arXiv 2412.13147 ↗First posted 17 Dec 2024cs.AI cs.CLcited by 65

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The rapid advancement of large language models (LLMs) has shown remarkable progress in complex reasoning tasks. However, a significant disparity exists between benchmark performances and real-world applications. We attribute this gap primarily to current evaluation protocols and metrics, which inadequately capture the full spectrum of LLM capabilities, especially in complex reasoning tasks where both accuracy and consistency are essential. In this paper, we introduce G-Pass@k, a novel evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model's performance potential and its stability. Through extensive experiments on various public and newly constructed benchmarks, we employ G-Pass@k in conjunction with state-of-the-art large language models to provide comprehensive insights into their potential capabilities and operational consistency. Our findings reveal a significant opportunity to enhance the realistic reasoning abilities of LLMs, underscoring the necessity for more robust evaluation metrics.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 21 of its 41 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

21 of 41 references.

References · 21

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  12. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
9 more
  1. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 197
  2. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  3. 2019SPoC: Search-based Pseudocode to Codecited by 176
  4. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  5. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 119
  6. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 105
  7. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  8. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  9. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 43; 2026: 22.

65 papers.

Sort:

By area:

  1. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  2. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  3. 2025Efficient Reasoning Models: A Surveycited by 95
  4. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  5. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 82
  6. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 55
  7. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 52
  8. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  9. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  10. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 38
  11. 2025R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluationcited by 34
  12. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 24
38 more
  1. 2025Solving Inequality Proofs with Large Language Modelscited by 21
  2. 2025InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scalingcited by 20
  3. 2025LLMs Provide Unstable Answers to Legal Questionscited by 19
  4. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  5. 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMscited by 14
  6. 2025No Need for Explanations: LLMs can implicitly learn from mistakes in-contextcited by 13
  7. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  8. 2025CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challengescited by 11
  9. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrowcited by 11
  10. 2025Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMscited by 10
  11. 2025Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoningcited by 10
  12. 2025SafeLawBench: Towards Safe Alignment of Large Language Modelscited by 9
  13. 2025Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundariescited by 9
  14. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoningcited by 9
  15. 2025Solve-Detect-Verify: Inference-Time Scaling with Flexible Generative Verifiercited by 8
  16. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  17. 2025Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Modelscited by 6
  18. 2025SCI-Verifier: Scientific Verifier with Thinkingcited by 6
  19. 2025Generalized Parallel Scaling with Interdependent Generationscited by 6
  20. 2025ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoningcited by 6
  21. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 6
  22. 2025dots.llm1 Technical Reportcited by 5
  23. 2025RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Modelscited by 5
  24. 2026LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?cited by 5
  25. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  26. 2025PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoningcited by 4
  27. 2025Rectifying LLM Thought from Lens of Optimizationcited by 4
  28. 2025HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworkscited by 4
  29. 2026EternalMath: A Living Benchmark of Frontier Mathematics that Evolves with Human Discoverycited by 4
  30. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Modelscited by 4
  31. 2025Integrating Expert Knowledge into Logical Programs via LLMscited by 3
  32. 2025Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflectioncited by 3
  33. 2026On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generationcited by 3
  34. 2026Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMscited by 3
  35. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Trainingcited by 3
  36. 2026Federation over Text: Insight Sharing for Multi-Agent Reasoningcited by 3
  37. 2025Swarm Intelligence Enhanced Reasoning: A Density-Driven Framework for LLM-Based Multi-Agent Optimizationcited by 2
  38. 2025Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspectivecited by 2

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  2. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
  3. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  4. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  5. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
  6. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  7. 2026Large Language Model Reasoning Failures
  8. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes