mlsanityData to 14 Oct 2026

Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning

arXiv 2509.07238 ↗First posted 8 Sep 2025cs.LG cs.AIcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

This paper presents a practical investigation into fine-tuning model parameters for mathematical reasoning tasks through experimenting with various configurations including randomness control, reasoning depth, and sampling strategies, careful tuning demonstrates substantial improvements in efficiency as well as performance. A holistically optimized framework is introduced for five state-of-the-art models on mathematical reasoning tasks, exhibiting significant performance boosts while maintaining solution correctness. Through systematic parameter optimization across Qwen2.5-72B, Llama-3.1-70B, DeepSeek-V3, Mixtral-8x22B, and Yi-Lightning, consistent efficiency gains are demonstrated with 100% optimization success rate. The methodology achieves an average 29.4% reduction in computational cost and 23.9% improvement in inference speed across all tested models. This framework systematically searches parameter spaces including temperature (0.1-0.5), reasoning steps (4-12), planning periods (1-4), and nucleus sampling (0.85-0.98), determining optimal configurations through testing on mathematical reasoning benchmarks. Critical findings show that lower temperature regimes (0.1-0.4) and reduced reasoning steps (4-6) consistently enhance efficiency without compromising accuracy. DeepSeek-V3 achieves the highest accuracy at 98%, while Mixtral-8x22B delivers the most cost-effective performance at 361.5 tokens per accurate response. Key contributions include: (1) the first comprehensive optimization study for five diverse SOTA models in mathematical reasoning, (2) a standardized production-oriented parameter optimization framework, (3) discovery of universal optimization trends applicable across model architectures, and (4) production-ready configurations with extensive performance characterization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 22 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

20 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning14 Oct
  2. 2024LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought9 May

Other references · 18

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  3. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 67
  4. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 64
  5. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoningcited by 40
  6. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  7. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  8. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 21
  9. 2024General Purpose Verification for Chain of Thought Promptingcited by 17
  10. 2025Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoningcited by 17
  11. 2023Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answeringcited by 14
  12. 2025Temporalizing Confidence: Evaluation of Chain-of-Thought Reasoning with Signal Temporal Logiccited by 11
6 more
  1. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  2. 2024Optimization Hyper-parameter Laws for Large Language Modelscited by 7
  3. 2025Scalable LLM Reasoning Acceleration with Low-rank Distillationcited by 7
  4. 2025Efficient LLM Collaboration via Planningcited by 7
  5. 2024A Post-Training Enhanced Optimization Approach for Small Language Modelscited by 1
  6. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
  2. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  3. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Models
  4. 2025A Survey on Large Language Models for Mathematical Reasoning
  5. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  6. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time
  7. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
  8. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data