# ChatGPT as a Math Questioner? Evaluating ChatGPT on Generating Pre-university Math Questions

arXiv: https://arxiv.org/abs/2312.01661 · HTML: https://mlsanity.com/paper/2312.01661/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression](https://mlsanity.com/paper/2212.02746/) (2022)
- [MixQG: Neural Question Generation with Mixed Answer Types](https://mlsanity.com/paper/2110.08175/) (2021)
- [Math Word Problem Generation with Mathematical Consistency and Problem Context Constraints](https://mlsanity.com/paper/2109.04546/) (2021)
- [Mathematical Word Problem Generation from Commonsense Knowledge Graph and Equations](https://mlsanity.com/paper/2010.06196/) (2020)
- [GTM: A Generative Triple-Wise Model for Conversational Question Generation](https://mlsanity.com/paper/2106.03635/) (2021)
- [Reasoning Circuits: Few-shot Multihop Question Generation with Structured Rationales](https://mlsanity.com/paper/2211.08466/) (2022)

## Built on by (most cited first, 8 of 8)

- [Curriculum Demonstration Selection for In-Context Learning](https://mlsanity.com/paper/2411.18126/) (2024)
- [Efficient Evaluation of Large Language Models via Collaborative Filtering](https://mlsanity.com/paper/2504.08781/) (2025)
- [When Not to Answer: Evaluating Prompts on GPT Models for Effective Abstention in Unanswerable Math Word Problems](https://mlsanity.com/paper/2410.13029/) (2024)
- [DOP: Diagnostic-Oriented Prompting for Large Language Models in Mathematical Correction](https://mlsanity.com/paper/2405.12100/) (2024)
- [The ORCA Benchmark: Evaluating Real-World Calculation Accuracy in Large Language Models](https://mlsanity.com/paper/2511.02589/) (2025)
- [EduAgentQG: Multi-Agent Personalized Mathematics Question Generation with Explicit Diversity and Objective-Aware Evaluation](https://mlsanity.com/paper/2511.11635/) (2025)
- [Large Language Models for Education and Research: An Empirical and User Survey-based Analysis](https://mlsanity.com/paper/2512.08057/) (2025)
- [A self-evolving multi-role collaborative framework with fine-grained difficulty guidance for innovative mathematical problem generation](https://mlsanity.com/paper/2601.11792/) (2026)

## Abstract (arXiv)

Mathematical questioning is crucial for assessing students problem-solving skills. Since manually creating such questions requires substantial effort, automatic methods have been explored. Existing state-of-the-art models rely on fine-tuning strategies and struggle to generate questions that heavily involve multiple steps of logical and arithmetic reasoning. Meanwhile, large language models(LLMs) such as ChatGPT have excelled in many NLP tasks involving logical and arithmetic reasoning. Nonetheless, their applications in generating educational questions are underutilized, especially in the field of mathematics. To bridge this gap, we take the first step to conduct an in-depth analysis of ChatGPT in generating pre-university math questions. Our analysis is categorized into two main settings: context-aware and context-unaware. In the context-aware setting, we evaluate ChatGPT on existing math question-answering benchmarks covering elementary, secondary, and ternary classes. In the context-unaware setting, we evaluate ChatGPT in generating math questions for each lesson from pre-university math curriculums that we crawl. Our crawling results in TopicMath, a comprehensive and novel collection of pre-university math curriculums collected from 121 math topics and 428 lessons from elementary, secondary, and tertiary classes. Through this analysis, we aim to provide insight into the potential of ChatGPT as a math questioner.