mlsanityData to 7 Oct 2026

Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning

arXiv 2403.02333 ↗First posted 4 Mar 2024cs.CL cs.AIcited by 58

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have shown great potential in complex reasoning tasks, yet their performance is often hampered by the scarcity of high-quality and reasoning-focused training datasets. Addressing this challenge, we propose Key-Point-Driven Data Synthesis (KPDDS), a novel data synthesis framework that synthesizes question-answer pairs by leveraging key points and exemplar practices from authentic data sources. KPDDS ensures the generation of novel questions with rigorous quality control and substantial scalability. As a result, we present KPMath, an extensive synthetic dataset tailored for mathematical reasoning, comprising over 800K question-answer pairs. Utilizing KPMath and augmenting it with additional reasoning-intensive corpora, we create the comprehensive KPMath-Plus dataset. The Qwen1.5-72B model, fine-tuned on KPMath-Plus, achieves 87.0% PASS@1 accuracy on GSM8K and 58.3% on MATH, surpassing competitors in the 7B to 70B range and best commercial models like GPT-4 across multiple math reasoning datasets.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 91 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

53 of 91 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data14 Feb
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  3. 2024Augmenting Math Word Problems via Iterative Question Composing17 Jan
  4. 2023Learning From Mistakes Makes LLM Better Reasoner31 Oct
  5. 2023Llemma: An Open Language Model For Mathematics16 Oct
  6. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text10 Oct
  7. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  9. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification15 Aug
  10. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning4 Jun
  11. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning29 Sep
  12. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 41

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Mistral 7Bcited by 3,065
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2023PaLM 2 Technical Reportcited by 1,154
29 more
  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  6. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  7. 2022PAL: Program-aided Language Modelscited by 635
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  9. 2023Textbooks Are All You Needcited by 561
  10. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  11. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  12. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  13. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  14. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  15. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  16. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  17. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  18. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96
  19. 2023Cumulative Reasoning with Large Language Modelscited by 94
  20. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 88
  21. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  22. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  23. 2023Contrastive Chain-of-Thought Promptingcited by 52
  24. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Modelscited by 33
  25. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuningcited by 28
  26. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  27. 2023LANS: A Layout-Aware Neural Solver for Plane Geometry Problemcited by 23
  28. 2023S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Modelscited by 18
  29. 2024Using Left and Right Brains Together: Towards Vision and Language Planningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 28; 2026: 6.

58 papers.

Sort:

By area:

  1. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  4. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  5. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  6. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  7. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  8. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  9. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 53
  10. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  11. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 49
  12. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
38 more
  1. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  2. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 39
  3. 2024AI-Assisted Generation of Difficult Math Questionscited by 37
  4. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  5. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  6. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  7. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  8. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 22
  9. 2025PromptCoT: Synthesizing Olympiad-level Problems for Mathematical Reasoning in Large Language Modelscited by 20
  10. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 20
  11. 2024DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoningcited by 19
  12. 2024Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoningcited by 15
  13. 2024LogicPro: Improving Complex Logical Reasoning via Program-Guided Learningcited by 14
  14. 2024MIND: Math Informed syNthetic Dialogues for Pretraining LLMscited by 14
  15. 2024Exploring the Mystery of Influential Data for Mathematical Reasoningcited by 13
  16. 2024Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoningcited by 13
  17. 2025Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLMcited by 13
  18. 2025SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answerscited by 11
  19. 2025Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoningcited by 10
  20. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  21. 2026Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utilitycited by 8
  22. 2024Task Oriented In-Domain Data Augmentationcited by 6
  23. 2025MathDebugger: Detecting and Diagnosing Errors in Synthetic Mathematical Datacited by 6
  24. 2025TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioningcited by 6
  25. 2025DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoningcited by 6
  26. 2025ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoningcited by 6
  27. 2024Compound-QA: A Benchmark for Evaluating LLMs on Compound Questionscited by 4
  28. 2025A Survey of LLM × DATAcited by 4
  29. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  30. 2024Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generationcited by 3
  31. 2025DAST: Difficulty-Aware Self-Training on Large Language Modelscited by 3
  32. 2024GRIP: A Graph-Based Reasoning Instruction Producercited by 2
  33. 2024DS2-ABSA: Dual-Stream Data Synthesis with Label Refinement for Few-Shot Aspect-Based Sentiment Analysiscited by 2
  34. 2024NILE: Internal Consistency Alignment in Large Language Modelscited by 2
  35. 2025FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipelinecited by 2
  36. 2025LEKA:LLM-Enhanced Knowledge Augmentationcited by 1
  37. 2025RV-Syn: Rational and Verifiable Mathematical Reasoning Data Synthesis based on Structured Function Librarycited by 1
  38. 2025Can Structured Templates Facilitate LLMs in Tackling Harder Tasks? : An Exploration of Scaling Laws by Difficultycited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model
  2. 2025Arrows of Math Reasoning Data Synthesis for Large Language Models: Diversity, Complexity and Correctness
  3. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs
  4. 2025More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning
  5. 2025PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data
  6. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
  7. 2024LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning
  8. 2025Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages