# Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems

arXiv: https://arxiv.org/abs/2410.00151 · HTML: https://mlsanity.com/paper/2410.00151/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 24 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process](https://mlsanity.com/paper/2407.20311/) (2024)
- [Premise Order Matters in Reasoning with Large Language Models](https://mlsanity.com/paper/2402.08939/) (2024)
- [On Leakage of Code Generation Evaluation Datasets](https://mlsanity.com/paper/2407.07565/) (2024)

## Built on by (most cited first, 7 of 7)

- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring](https://mlsanity.com/paper/2505.23575/) (2025)
- [SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens](https://mlsanity.com/paper/2510.24940/) (2025)
- [StepMathAgent: A Step-Wise Agent for Evaluating Mathematical Processes through Tree-of-Error](https://mlsanity.com/paper/2503.10105/) (2025)
- [Large Language Models Don't Make Sense of Word Problems. A Scoping Review from a Mathematics Education Perspective](https://mlsanity.com/paper/2506.24006/) (2025)
- [Advancing AI Research Assistants with Expert-Involved Learning](https://mlsanity.com/paper/2505.04638/) (2025)
- [Law And Order: Tax Law Autoformalization](https://mlsanity.com/paper/2610.02792/) (2026)

## Abstract (arXiv)

Benchmarks are critical for measuring Large Language Model (LLM) reasoning capabilities. Some benchmarks have even become the de facto indicator of such capabilities. However, as LLM reasoning capabilities improve, existing widely-used benchmarks such as GSM8K marginally encapsulate model reasoning differentials - most state-of-the-art models for example achieve over 94% accuracy on the GSM8K dataset (paperwithcode, 2024). While constructing harder benchmarks is possible, their creation is often manual, expensive, and unscalable. As such, we present Scheherazade, an automated approach to produce large quantities of challenging mathematical reasoning benchmarks by logically chaining a small starting set of problems. We propose two different chaining methods, forward chaining and backward chaining, which include randomized branching techniques to generate complex reasoning problems. We apply Scheherazade on GSM8K to create GSM8K-Scheherazade and evaluate 3 frontier LLMs and OpenAI's o1-preview on it. We show that while other frontier models' performance declines precipitously at only a few questions chained, our evaluation suggests o1-preview's performance persists, with the flagship OpenAI model the only one to perform better at backward reasoning. Our data and code are available at https://github.com/YoshikiTakashima/scheherazade-code-data.