# Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems

arXiv: https://arxiv.org/abs/1609.07197 · HTML: https://mlsanity.com/paper/1609.07197/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2016.

## Built on by (most cited first, 28 of 28)

- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark](https://mlsanity.com/paper/2405.12209/) (2024)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers](https://mlsanity.com/paper/2109.00799/) (2021)
- [An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)](https://mlsanity.com/paper/2302.13814/) (2023)
- [Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering](https://mlsanity.com/paper/2402.11194/) (2024)
- [Mathematical Language Models: A Survey](https://mlsanity.com/paper/2312.07622/) (2023)
- [Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solvers](https://mlsanity.com/paper/2205.15683/) (2022)
- [Unbiased Math Word Problems Benchmark for Mitigating Solving Bias](https://mlsanity.com/paper/2205.08108/) (2022)
- [Solving for X and Beyond: Can Large Language Models Solve Complex Math Problems with More-Than-Two Unknowns?](https://mlsanity.com/paper/2407.05134/) (2024)
- [Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems](https://mlsanity.com/paper/2211.07455/) (2022)
- [Enhancing Neural Mathematical Reasoning by Abductive Combination with Symbolic Library](https://mlsanity.com/paper/2203.14487/) (2022)
- [Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queries](https://mlsanity.com/paper/2308.11189/) (2023)
- [Seeking Diverse Reasoning Logic: Controlled Equation Expression Generation for Solving Math Word Problems](https://mlsanity.com/paper/2209.10310/) (2022)
- [Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard](https://mlsanity.com/paper/2305.18618/) (2023)
- [Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends](https://mlsanity.com/paper/2401.07518/) (2024)
- [EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMs](https://mlsanity.com/paper/2510.17389/) (2025)
- [STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework](https://mlsanity.com/paper/2506.01531/) (2025)
- [DISK: Domain-constrained Instance Sketch for Math Word Problem Generation](https://mlsanity.com/paper/2204.04686/) (2022)
- [Textual Enhanced Contrastive Learning for Solving Math Word Problems](https://mlsanity.com/paper/2211.16022/) (2022)
- [Error Detection and Correction for Interpretable Mathematics in Large Language Models](https://mlsanity.com/paper/2508.03500/) (2025)
- [Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving](https://mlsanity.com/paper/2404.03938/) (2024)
- [Program of Equations Thoughts to Solve Algebra Word Problems](https://mlsanity.com/paper/2505.20170/) (2025)
- [From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities](https://mlsanity.com/paper/2608.26950/) (2026)

## Abstract (arXiv)

We propose a new evaluation for automatic solvers for algebra word problems, which can identify mistakes that existing evaluations overlook. Our proposal is to evaluate such solvers using derivations, which reflect how an equation system was constructed from the word problem. To accomplish this, we develop an algorithm for checking the equivalence between two derivations, and show how derivation an- notations can be semi-automatically added to existing datasets. To make our experiments more comprehensive, we include the derivation annotation for DRAW-1K, a new dataset containing 1000 general algebra word problems. In our experiments, we found that the annotated derivations enable a more accurate evaluation of automatic solvers than previously used metrics. We release derivation annotations for over 2300 algebra word problems for future evaluations.