mlsanityData to 7 Oct 2026

TheoremQA: A Theorem-driven Question Answering dataset

arXiv 2305.12524 ↗First posted 21 May 2023cs.CL cs.AIcited by 189

Not on the map. Why?

Cited by 84 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. However, their capabilities to solve more challenging math problems which require domain-specific knowledge (i.e. theorem) have yet to be investigated. In this paper, we introduce TheoremQA, the first theorem-driven question-answering dataset designed to evaluate AI models' capabilities to apply theorems to solve challenging science problems. TheoremQA is curated by domain experts containing 800 high-quality questions covering 350 theorems (e.g. Taylor's theorem, Lagrange's theorem, Huffman coding, Quantum Theorem, Elasticity Theorem, etc) from Math, Physics, EE&CS, and Finance. We evaluate a wide spectrum of 16 large language and code models with different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. We found that GPT-4's capabilities to solve these problems are unparalleled, achieving an accuracy of 51% with Program-of-Thoughts Prompting. All the existing open-sourced models are below 15%, barely surpassing the random-guess baseline. Given the diversity and broad coverage of TheoremQA, we believe it can be used as a better benchmark to evaluate LLMs' capabilities to solve challenging science problems. The data and code are released in https://github.com/wenhuchen/TheoremQA.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

51 of 59 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  15. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  16. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  17. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  18. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  19. 2023StarCoder: may the source be with you!cited by 790
  20. 2022Galactica: A Large Language Model for Sciencecited by 736
  21. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  22. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  23. 2022PAL: Program-aided Language Modelscited by 635
  24. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  25. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  26. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  27. 2016Solving General Arithmetic Word Problemscited by 353
  28. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  29. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  30. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  31. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  32. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  33. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  34. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  35. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  36. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  37. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  38. 2016Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problemscited by 28

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 44; 2025: 74; 2026: 53.

189 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  8. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  9. 2023Detecting Language Model Attacks with Perplexitycited by 411
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  11. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  12. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
38 more
  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  4. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  5. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  6. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  7. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  8. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  9. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  10. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  11. 2024A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalistcited by 137
  12. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  13. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  14. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  15. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  16. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  17. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 87
  18. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  19. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  20. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  21. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  22. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 69
  23. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  24. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  25. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 61
  26. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
  27. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  28. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  29. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  30. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 56
  31. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  32. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52
  33. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  34. 2025FineVision: Open Data Is All You Needcited by 51
  35. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  36. 2025A Survey of Large Language Model Agents for Question Answeringcited by 46
  37. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
  38. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44

50 most cited of 189 All 189 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Let's Verify Math Questions Step by Step
  2. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time
  3. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
  4. 2025Saturation-Driven Dataset Generation for LLM Mathematical Reasoning in the TPTP Ecosystem
  5. 2026More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
  6. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch
  7. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs
  8. 2025GraphMind: Theorem Selection and Conclusion Generation Framework with Dynamic GNN for LLM Reasoning