# MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving

arXiv: https://arxiv.org/abs/2107.13435 · HTML: https://mlsanity.com/paper/2107.13435/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 28 Jul 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problems](https://mlsanity.com/paper/2009.11506/) (2020)
- [Translating a Math Word Problem to an Expression Tree](https://mlsanity.com/paper/1811.05632/) (2018)
- [Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension](https://mlsanity.com/paper/1909.00109/) (2019)
- [Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems](https://mlsanity.com/paper/1811.00720/) (2018)
- [Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks](https://mlsanity.com/paper/2107.01431/) (2021)
- [Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems](https://mlsanity.com/paper/2010.06823/) (2020)
- [Learning by Fixing: Solving Math Word Problems with Weak Supervision](https://mlsanity.com/paper/2012.10582/) (2020)
- [Investigating Math Word Problems using Pretrained Multilingual Language Models](https://mlsanity.com/paper/2105.08928/) (2021)
- [Mathematical Reasoning in Latent Space](https://mlsanity.com/paper/1909.11851/) (2019)
- [SMART: A Situation Model for Algebra Story Problems via Attributed Grammar](https://mlsanity.com/paper/2012.14011/) (2020)
- [Text2Math: End-to-end Parsing Text into Math Expressions](https://mlsanity.com/paper/1910.06571/) (2019)
- [Quantity Tagger: A Latent-Variable Sequence Labeling Approach to Solving Addition-Subtraction Word Problems](https://mlsanity.com/paper/1909.00176/) (2019)

## Built on by (most cited first, 30 of 32)

- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level](https://mlsanity.com/paper/2112.15594/) (2021)
- [A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram](https://mlsanity.com/paper/2302.11097/) (2023)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training](https://mlsanity.com/paper/2310.02227/) (2023)
- [Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation](https://mlsanity.com/paper/2305.14386/) (2023)
- [Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering](https://mlsanity.com/paper/2402.11194/) (2024)
- [Mathematical Language Models: A Survey](https://mlsanity.com/paper/2312.07622/) (2023)
- [Learning by Analogy: Diverse Questions Generation in Math Word Problem](https://mlsanity.com/paper/2306.09064/) (2023)
- [MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning](https://mlsanity.com/paper/2307.07951/) (2023)
- [Tree-Based Representation and Generation of Natural and Mathematical Language](https://mlsanity.com/paper/2302.07974/) (2023)
- [Learning Multi-Step Reasoning by Solving Arithmetic Tasks](https://mlsanity.com/paper/2306.01707/) (2023)
- [A Survey on Mathematical Reasoning and Optimization with Large Language Models](https://mlsanity.com/paper/2503.17726/) (2025)
- [FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Models](https://mlsanity.com/paper/2403.07747/) (2024)
- [Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning](https://mlsanity.com/paper/2505.21354/) (2025)
- [Generalizing Math Word Problem Solvers via Solution Diversification](https://mlsanity.com/paper/2212.00833/) (2022)
- [Non-Autoregressive Math Word Problem Solver with Unified Tree Structure](https://mlsanity.com/paper/2305.04556/) (2023)
- [Solving Math Word Problem with Problem Type Classification](https://mlsanity.com/paper/2308.13844/) (2023)
- [Analogical Math Word Problems Solving with Enhanced Problem-Solution Association](https://mlsanity.com/paper/2212.00837/) (2022)
- [FERMAT: An Alternative to Accuracy for Numerical Reasoning](https://mlsanity.com/paper/2305.17491/) (2023)
- [Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventions](https://mlsanity.com/paper/2404.18384/) (2024)
- [Empowering Bengali Education with AI: Solving Bengali Math Word Problems through Transformer Models](https://mlsanity.com/paper/2501.02599/) (2025)
- [From Large to Tiny: Distilling and Refining Mathematical Expertise for Math Word Problems with Weakly Supervision](https://mlsanity.com/paper/2403.14390/) (2024)
- [Transparent AI for Mathematics: Transformer-Based Large Language Models for Mathematical Entity Relationship Extraction with XAI](https://mlsanity.com/paper/2603.06348/) (2026)
- [FGeo-TP: A Language Model-Enhanced Solver for Geometry Problems](https://mlsanity.com/paper/2402.09047/) (2024)
- [Foundation of Intelligence: Review of Math Word Problems from Human Cognition Perspective](https://mlsanity.com/paper/2510.21999/) (2025)
- [Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery](https://mlsanity.com/paper/2606.08728/) (2026)
- [Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning](https://mlsanity.com/paper/2305.07475/) (2023)
- [Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving](https://mlsanity.com/paper/2404.03938/) (2024)
- [A Theoretical Review on Solving Algebra Problems](https://mlsanity.com/paper/2411.00031/) (2024)

## Abstract (arXiv)

Math word problem (MWP) solving faces a dilemma in number representation learning. In order to avoid the number representation issue and reduce the search space of feasible solutions, existing works striving for MWP solving usually replace real numbers with symbolic placeholders to focus on logic reasoning. However, different from common symbolic reasoning tasks like program synthesis and knowledge graph reasoning, MWP solving has extra requirements in numerical reasoning. In other words, instead of the number value itself, it is the reusable numerical property that matters more in numerical reasoning. Therefore, we argue that injecting numerical properties into symbolic placeholders with contextualized representation learning schema can provide a way out of the dilemma in the number representation issue here. In this work, we introduce this idea to the popular pre-training language model (PLM) techniques and build MWP-BERT, an effective contextual number representation PLM. We demonstrate the effectiveness of our MWP-BERT on MWP solving and several MWP-specific understanding tasks on both English and Chinese benchmarks.