mlsanityData to 14 Oct 2026

ARB: Advanced Reasoning Benchmark for Large Language Models

arXiv 2307.13692 ↗First posted 25 Jul 2023cs.CL cs.LGcited by 60

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance on various quantitative reasoning and knowledge benchmarks. However, many of these benchmarks are losing utility as LLMs get increasingly high scores, despite not yet reaching expert performance in these domains. We introduce ARB, a novel benchmark composed of advanced reasoning problems in multiple fields. ARB presents a more challenging test than prior benchmarks, featuring problems in mathematics, physics, biology, chemistry, and law. As a subset of ARB, we introduce a challenging set of math and physics problems which require advanced symbolic reasoning and domain knowledge. We evaluate recent models such as GPT-4 and Claude on ARB and demonstrate that current models score well below 50% on more demanding tasks. In order to improve both automatic and assisted evaluation capabilities, we introduce a rubric-based evaluation approach, allowing GPT-4 to score its own intermediate reasoning steps. Further, we conduct a human evaluation of the symbolic subset of ARB, finding promising agreement between annotators and GPT-4 rubric evaluation scores.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 72 references that are in the map. First posted on arXiv 25 Jul 2023.

Builds on

22 of 72 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
9 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  3. 2023GPTScore: Evaluate as You Desirecited by 361
  4. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 171
  5. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 163
  6. 2019On the Measure of Intelligencecited by 144
  7. 2022GPT Takes the Bar Examcited by 65
  8. 2023An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)cited by 31
  9. 2022From Human Days to Machine Seconds: Automatically Answering and Generating Machine Learning Final Examscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 26; 2025: 14; 2026: 10.

60 papers.

Sort:

By area:

  1. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  2. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  3. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  4. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  5. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  6. 2024Nemotron-4 340B Technical Reportcited by 144
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  8. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  9. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 115
  10. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
  11. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
  12. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 89
38 more
  1. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  2. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  3. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  4. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  5. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  6. 2025Reasoning Language Models: A Blueprintcited by 42
  7. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  8. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 34
  9. 2024HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematicscited by 32
  10. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languagescited by 28
  11. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  12. 2025FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levelscited by 27
  13. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 24
  14. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  15. 2024Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilitiescited by 22
  16. 2024Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventionscited by 20
  17. 2024Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competitioncited by 20
  18. 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Modelscited by 19
  19. 2024Causal Evaluation of Language Modelscited by 18
  20. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14
  21. 2023GLoRE: Evaluating Logical Reasoning of Large Language Modelscited by 13
  22. 2024Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoningcited by 13
  23. 2024Evaluating LLM Reasoning in the Operations Research Domain with ORQAcited by 12
  24. 2024Data for Mathematical Copilots: Better Ways of Presenting Proofs for Machine Learningcited by 11
  25. 2025Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMscited by 11
  26. 2025ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modelingcited by 10
  27. 2024ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoningcited by 9
  28. 2024UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughtscited by 8
  29. 2026Riemann-Bench: A Benchmark for Moonshot Mathematicscited by 8
  30. 2024RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasonerscited by 6
  31. 2026Learning from Synthetic Data Improves Multi-hop Reasoningcited by 6
  32. 2025HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Classcited by 5
  33. 2025Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoningcited by 5
  34. 2026LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketchescited by 5
  35. 2024What is the best model? Application-driven Evaluation for Large Language Modelscited by 4
  36. 2023ArcGPT: A Large Language Model Tailored for Real-world Archival Applicationscited by 3
  37. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  38. 2024RoMath: A Mathematical Reasoning Benchmark in Romaniancited by 2

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  2. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models
  3. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
  4. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  5. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models
  6. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models
  7. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?
  8. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance