mlsanityData to 7 Oct 2026

Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models

arXiv 2305.15074 ↗First posted 24 May 2023cs.CL cs.AIcited by 76

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

The performance of large language models (LLMs) on existing reasoning benchmarks has significantly improved over the past years. In response, we present JEEBench, a considerably more challenging benchmark dataset for evaluating the problem solving abilities of LLMs. We curate 515 challenging pre-engineering mathematics, physics and chemistry problems from the highly competitive IIT JEE-Advanced exam. Long-horizon reasoning on top of deep in-domain knowledge is essential for solving problems in this benchmark. Our evaluation on various open-source and proprietary models reveals that the highest performance, even after using techniques like self-consistency, self-refinement and chain-of-thought prompting, is less than 40%. The typical failure modes of GPT-4, the best model, are errors in algebraic manipulation, difficulty in grounding abstract concepts into mathematical equations accurately and failure in retrieving relevant domain-specific concepts. We also observe that by mere prompting, GPT-4 is unable to assess risk introduced by negative marking for incorrect answers. For this, we develop a post-hoc confidence-thresholding method over self-consistency, which enables effective response selection. We hope that our challenging benchmark will guide future re-search in problem-solving using LLMs.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 32 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

19 of 32 references.

References · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
7 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  2. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  3. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  4. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  5. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  6. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
  7. 2020Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problemscited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 20; 2025: 27; 2026: 19.

76 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  5. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  6. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  7. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  8. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  9. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  10. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  11. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 45
  12. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
38 more
  1. 2024Are large language models superhuman chemists?cited by 34
  2. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  3. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  4. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physicscited by 33
  5. 2024HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematicscited by 32
  6. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  7. 2026Maximum Likelihood Reinforcement Learningcited by 29
  8. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  9. 2024Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agentscited by 22
  10. 2024MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Promptingcited by 15
  11. 2025SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Modelscited by 15
  12. 2024Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awarenesscited by 14
  13. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  14. 2023Neural Task Synthesis for Visual Programmingcited by 13
  15. 2024PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbationscited by 13
  16. 2024Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoningcited by 13
  17. 2023MaScQA: A Question Answering Dataset for Investigating Materials Science Knowledge of Large Language Modelscited by 11
  18. 2024Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistantscited by 10
  19. 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solvingcited by 10
  20. 2024NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Modelscited by 9
  21. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 9
  22. 2025Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectoriescited by 8
  23. 2026Riemann-Bench: A Benchmark for Moonshot Mathematicscited by 8
  24. 2024textttLM^texttt2: A Simple Society of Language Models Solves Complex Reasoningcited by 7
  25. 2023Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Modelscited by 6
  26. 2025Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Modelscited by 6
  27. 2025QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistrycited by 5
  28. 2025DiSRouter: Distributed Self-Routing for LLM Selectionscited by 5
  29. 2026MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrievalcited by 5
  30. 2024A Comprehensive Evaluation on Event Reasoning of Large Language Modelscited by 4
  31. 2024IntellBot: Retrieval Augmented LLM Chatbot for Cyber Threat Knowledge Deliverycited by 4
  32. 2024ElectroVizQA: How well do Multi-modal LLMs perform in Electronics Visual Question Answering?cited by 4
  33. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  34. 2024Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programmingcited by 3
  35. 2024Synergistic Simulations: Multi-Agent Problem Solving with Large Language Modelscited by 3
  36. 2025DebateBench: A Challenging Long Context Reasoning Benchmark For Large Language Modelscited by 3
  37. 2025RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoningcited by 3
  38. 2025Adversarial Training for Process Reward Modelscited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluation of LLMs for mathematical problem solving
  2. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  3. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  4. 2024LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs
  5. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  6. 2025Large Language Models and Mathematical Reasoning Failures
  7. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models
  8. 2025Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?