# Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems

arXiv: https://arxiv.org/abs/2110.08464 · HTML: https://mlsanity.com/paper/2110.08464/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach](https://mlsanity.com/paper/2010.07835/) (2020)
- [Improving BERT with Syntax-aware Local Attention](https://mlsanity.com/paper/2012.15150/) (2020)
- [Adversarial Contrastive Estimation](https://mlsanity.com/paper/1805.03642/) (2018)
- [Investigating Math Word Problems using Pretrained Multilingual Language Models](https://mlsanity.com/paper/2105.08928/) (2021)

## Built on by (most cited first, 30 of 30)

- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram](https://mlsanity.com/paper/2302.11097/) (2023)
- [Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction](https://mlsanity.com/paper/2203.10316/) (2022)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation](https://mlsanity.com/paper/2305.14386/) (2023)
- [Mathematical Language Models: A Survey](https://mlsanity.com/paper/2312.07622/) (2023)
- [GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach](https://mlsanity.com/paper/2308.09267/) (2023)
- [A Symbolic Character-Aware Model for Solving Geometry Problems](https://mlsanity.com/paper/2308.02823/) (2023)
- [Learning by Analogy: Diverse Questions Generation in Math Word Problem](https://mlsanity.com/paper/2306.09064/) (2023)
- [Interpretable Math Word Problem Solution Generation Via Step-by-step Planning](https://mlsanity.com/paper/2306.00784/) (2023)
- [MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning](https://mlsanity.com/paper/2307.07951/) (2023)
- [Chaining Simultaneous Thoughts for Numerical Reasoning](https://mlsanity.com/paper/2211.16482/) (2022)
- [Multi-View Reasoning: Consistent Contrastive Learning for Math Word Problem](https://mlsanity.com/paper/2210.11694/) (2022)
- [A Survey on Mathematical Reasoning and Optimization with Large Language Models](https://mlsanity.com/paper/2503.17726/) (2025)
- [Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solvers](https://mlsanity.com/paper/2205.15683/) (2022)
- [Generalizing Math Word Problem Solvers via Solution Diversification](https://mlsanity.com/paper/2212.00833/) (2022)
- [Non-Autoregressive Math Word Problem Solver with Unified Tree Structure](https://mlsanity.com/paper/2305.04556/) (2023)
- [Neural Machine Translation for Mathematical Formulae](https://mlsanity.com/paper/2305.16433/) (2023)
- [Math Word Problem Solving by Generating Linguistic Variants of Problem Statements](https://mlsanity.com/paper/2306.13899/) (2023)
- [How to Leverage Digit Embeddings to Represent Numbers?](https://mlsanity.com/paper/2407.00894/) (2024)
- [Analogical Math Word Problems Solving with Enhanced Problem-Solution Association](https://mlsanity.com/paper/2212.00837/) (2022)
- [An Expression Tree Decoding Strategy for Mathematical Equation Generation](https://mlsanity.com/paper/2310.09619/) (2023)
- [Seeking Diverse Reasoning Logic: Controlled Equation Expression Generation for Solving Math Word Problems](https://mlsanity.com/paper/2209.10310/) (2022)
- [Solving Math Word Problems with Reexamination](https://mlsanity.com/paper/2310.09590/) (2023)
- [Textual Enhanced Contrastive Learning for Solving Math Word Problems](https://mlsanity.com/paper/2211.16022/) (2022)
- [MWPRanker: An Expression Similarity Based Math Word Problem Retriever](https://mlsanity.com/paper/2307.01240/) (2023)
- [Foundation of Intelligence: Review of Math Word Problems from Human Cognition Perspective](https://mlsanity.com/paper/2510.21999/) (2025)
- [A Theoretical Review on Solving Algebra Problems](https://mlsanity.com/paper/2411.00031/) (2024)
- [A Diversity-Enhanced Knowledge Distillation Model for Practical Math Word Problem Solving](https://mlsanity.com/paper/2501.03670/) (2025)

## Abstract (arXiv)

Math Word Problem (MWP) solving needs to discover the quantitative relationships over natural language narratives. Recent work shows that existing models memorize procedures from context and rely on shallow heuristics to solve MWPs. In this paper, we look at this issue and argue that the cause is a lack of overall understanding of MWP patterns. We first investigate how a neural network understands patterns only from semantics, and observe that, if the prototype equations are the same, most problems get closer representations and those representations apart from them or close to other prototypes tend to produce wrong solutions. Inspired by it, we propose a contrastive learning approach, where the neural network perceives the divergence of patterns. We collect contrastive examples by converting the prototype equation into a tree and seeking similar tree structures. The solving model is trained with an auxiliary objective on the collected examples, resulting in the representations of problems with similar prototypes being pulled closer. We conduct experiments on the Chinese dataset Math23k and the English dataset MathQA. Our method greatly improves the performance in monolingual and multilingual settings.