# A Diversity-Enhanced Knowledge Distillation Model for Practical Math Word Problem Solving

arXiv: https://arxiv.org/abs/2501.03670 · HTML: https://mlsanity.com/paper/2501.03670/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [Generate & Rank: A Multi-task Framework for Math Word Problems](https://mlsanity.com/paper/2109.03034/) (2021)
- [Translating a Math Word Problem to an Expression Tree](https://mlsanity.com/paper/1811.05632/) (2018)
- [Reasoning Like Program Executors](https://mlsanity.com/paper/2201.11473/) (2022)
- [Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction](https://mlsanity.com/paper/2203.10316/) (2022)
- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021)
- [Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks](https://mlsanity.com/paper/2107.01431/) (2021)
- [Dynamic Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2109.11295/) (2021)
- [Mapping to Declarative Knowledge for Word Problem Solving](https://mlsanity.com/paper/1712.09391/) (2017)
- [Variational Neural Machine Translation](https://mlsanity.com/paper/1605.07869/) (2016)
- [Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems](https://mlsanity.com/paper/2010.06823/) (2020)
- [MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving](https://mlsanity.com/paper/2107.13435/) (2021)
- [Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems](https://mlsanity.com/paper/2110.08464/) (2021)
- [Investigating Math Word Problems using Pretrained Multilingual Language Models](https://mlsanity.com/paper/2105.08928/) (2021)
- [Learning by Analogy: Diverse Questions Generation in Math Word Problem](https://mlsanity.com/paper/2306.09064/) (2023)
- [Chaining Simultaneous Thoughts for Numerical Reasoning](https://mlsanity.com/paper/2211.16482/) (2022)
- [Multi-View Reasoning: Consistent Contrastive Learning for Math Word Problem](https://mlsanity.com/paper/2210.11694/) (2022)
- [Non-Autoregressive Math Word Problem Solver with Unified Tree Structure](https://mlsanity.com/paper/2305.04556/) (2023)
- [Math Word Problem Solving by Generating Linguistic Variants of Problem Statements](https://mlsanity.com/paper/2306.13899/) (2023)
- [On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond](https://mlsanity.com/paper/2004.09189/) (2020)

## Abstract (arXiv)

Math Word Problem (MWP) solving is a critical task in natural language processing, has garnered significant research interest in recent years. Various recent studies heavily rely on Seq2Seq models and their extensions (e.g., Seq2Tree and Graph2Tree) to generate mathematical equations. While effective, these models struggle to generate diverse but counterpart solution equations, limiting their generalization across various math problem scenarios. In this paper, we introduce a novel Diversity-enhanced Knowledge Distillation (DivKD) model for practical MWP solving. Our approach proposes an adaptive diversity distillation method, in which a student model learns diverse equations by selectively transferring high-quality knowledge from a teacher model. Additionally, we design a diversity prior-enhanced student model to better capture the diversity distribution of equations by incorporating a conditional variational auto-encoder. Extensive experiments on {four} MWP benchmark datasets demonstrate that our approach achieves higher answer accuracy than strong baselines while maintaining high efficiency for practical applications.