mlsanityData to 14 Oct 2026

Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems

arXiv 1609.07197 ↗First posted 23 Sep 2016cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

We propose a new evaluation for automatic solvers for algebra word problems, which can identify mistakes that existing evaluations overlook. Our proposal is to evaluate such solvers using derivations, which reflect how an equation system was constructed from the word problem. To accomplish this, we develop an algorithm for checking the equivalence between two derivations, and show how derivation an- notations can be semi-automatically added to existing datasets. To make our experiments more comprehensive, we include the derivation annotation for DRAW-1K, a new dataset containing 1000 general algebra word problems. In our experiments, we found that the annotated derivations enable a more accurate evaluation of automatic solvers than previously used metrics. We release derivation annotations for over 2300 algebra word problems for future evaluations.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 5; 2024: 7; 2025: 4; 2026: 1.

28 papers.

Sort:

By area:

  1. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  2. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  3. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  4. 2024Large Language Models for Education: A Survey and Outlookcited by 188
  5. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  6. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 105
  7. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  8. 2021MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solverscited by 39
  9. 2023An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)cited by 31
  10. 2024Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answeringcited by 25
  11. 2023Mathematical Language Models: A Surveycited by 24
  12. 2022Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solverscited by 9
16 more
  1. 2022Unbiased Math Word Problems Benchmark for Mitigating Solving Biascited by 8
  2. 2024Solving for X and Beyond: Can Large Language Models Solve Complex Math Problems with More-Than-Two Unknowns?cited by 7
  3. 2022Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systemscited by 6
  4. 2022Enhancing Neural Mathematical Reasoning by Abductive Combination with Symbolic Librarycited by 4
  5. 2023Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queriescited by 4
  6. 2022Seeking Diverse Reasoning Logic: Controlled Equation Expression Generation for Solving Math Word Problemscited by 3
  7. 2023Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bardcited by 3
  8. 2024Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trendscited by 3
  9. 2025EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMscited by 3
  10. 2025STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Frameworkcited by 2
  11. 2022DISK: Domain-constrained Instance Sketch for Math Word Problem Generationcited by 1
  12. 2022Textual Enhanced Contrastive Learning for Solving Math Word Problemscited by 1
  13. 2025Error Detection and Correction for Interpretable Mathematics in Large Language Modelscited by 1
  14. 2024Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solvingcited by 0
  15. 2025Program of Equations Thoughts to Solve Algebra Word Problemscited by 0
  16. 2026From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilitiescited by 0