# Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning

arXiv: https://arxiv.org/abs/2509.07238 · HTML: https://mlsanity.com/paper/2509.07238/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 20 of 20)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023)
- [Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models](https://mlsanity.com/paper/2401.00625/) (2024)
- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning](https://mlsanity.com/paper/2502.18001/) (2025)
- [LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought](https://mlsanity.com/paper/2405.06705/) (2024)
- [BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models](https://mlsanity.com/paper/2404.02827/) (2024)
- [Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques](https://mlsanity.com/paper/2406.02500/) (2024)
- [Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models](https://mlsanity.com/paper/2409.09510/) (2024)
- [General Purpose Verification for Chain of Thought Prompting](https://mlsanity.com/paper/2405.00204/) (2024)
- [Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.09853/) (2025)
- [CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning](https://mlsanity.com/paper/2410.10336/) (2024)
- [Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answering](https://mlsanity.com/paper/2304.13911/) (2023)
- [Temporalizing Confidence: Evaluation of Chain-of-Thought Reasoning with Signal Temporal Logic](https://mlsanity.com/paper/2506.08243/) (2025)
- [Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning](https://mlsanity.com/paper/2410.08598/) (2024)
- [Optimization Hyper-parameter Laws for Large Language Models](https://mlsanity.com/paper/2409.04777/) (2024)
- [Scalable LLM Reasoning Acceleration with Low-rank Distillation](https://mlsanity.com/paper/2505.07861/) (2025)
- [Efficient LLM Collaboration via Planning](https://mlsanity.com/paper/2506.11578/) (2025)
- [A Post-Training Enhanced Optimization Approach for Small Language Models](https://mlsanity.com/paper/2411.02939/) (2024)
- [Systematic Evaluation of Optimization Techniques for Long-Context Language Models](https://mlsanity.com/paper/2508.00305/) (2025)

## Abstract (arXiv)

This paper presents a practical investigation into fine-tuning model parameters for mathematical reasoning tasks through experimenting with various configurations including randomness control, reasoning depth, and sampling strategies, careful tuning demonstrates substantial improvements in efficiency as well as performance. A holistically optimized framework is introduced for five state-of-the-art models on mathematical reasoning tasks, exhibiting significant performance boosts while maintaining solution correctness. Through systematic parameter optimization across Qwen2.5-72B, Llama-3.1-70B, DeepSeek-V3, Mixtral-8x22B, and Yi-Lightning, consistent efficiency gains are demonstrated with 100% optimization success rate. The methodology achieves an average 29.4% reduction in computational cost and 23.9% improvement in inference speed across all tested models. This framework systematically searches parameter spaces including temperature (0.1-0.5), reasoning steps (4-12), planning periods (1-4), and nucleus sampling (0.85-0.98), determining optimal configurations through testing on mathematical reasoning benchmarks. Critical findings show that lower temperature regimes (0.1-0.4) and reduced reasoning steps (4-6) consistently enhance efficiency without compromising accuracy. DeepSeek-V3 achieves the highest accuracy at 98%, while Mixtral-8x22B delivers the most cost-effective performance at 361.5 tokens per accurate response. Key contributions include: (1) the first comprehensive optimization study for five diverse SOTA models in mathematical reasoning, (2) a standardized production-oriented parameter optimization framework, (3) discovery of universal optimization trends applicable across model architectures, and (4) production-ready configurations with extensive performance characterization.