Testing GPT-4-o1-preview on math and science problems: A follow-up study
Not on the map. Why?
Abstract (arXiv)
In August 2023, Scott Aaronson and I reported the results of testing GPT4 with the Wolfram Alpha and Code Interpreter plug-ins over a collection of 105 original high-school level and college-level science and math problems (Davis and Aaronson, 2023). In September 2024, I tested the recently released model GPT-4o1-preview on the same collection. Overall I found that performance had significantly improved, but was still considerably short of perfect. In particular, problems that involve spatial reasoning are often stumbling blocks.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Oct 2024.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 4.
4 papers.
- 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
- 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
- 2025Can Theoretical Physics Research Benefit from Language Agents?cited by 9
- 2025Conflict-Aware Fusion: Mitigating Logic Inertia in Large Language Models via Structured Cognitive Priorscited by 5
- 2025Conflict-Aware Fusion: Mitigating Logic Inertia in Large Language Models via Structured Cognitive Priors6 Dec 2025
- 2025Can Theoretical Physics Research Benefit from Language Agents?6 Jun 2025
- 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Models28 Feb 2025
- 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models16 Jan 2025
Similar, not cited
Close in wording, no citation either way
- 2023How is ChatGPT's behavior changing over time?
- 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification
- 2025Benchmarking Large Language Models for Calculus Problem-Solving: A Comparative Analysis
- 2024An Eye for an AI: Evaluating GPT-4o's Visual Perception Skills and Geometric Reasoning Skills Using Computer Graphics Questions
- 2024Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Exam
- 2023Investigating the Effectiveness of ChatGPT in Mathematical Reasoning and Problem Solving: Evidence from the Vietnamese National High School Graduation Examination
- 2024OpenAI-o1 AB Testing: Does the o1 model really do good reasoning in math problem solving?
- 2023Mathematical Capabilities of ChatGPT