# Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning

arXiv: https://arxiv.org/abs/2502.07154 · HTML: https://mlsanity.com/paper/2502.07154/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 15 of 15)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning](https://mlsanity.com/paper/2402.06332/) (2024)
- [Preserving Diversity in Supervised Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2408.16673/) (2024)
- [Scaling Scaling Laws with Board Games](https://mlsanity.com/paper/2104.03113/) (2021)
- [A Simple Model of Inference Scaling Laws](https://mlsanity.com/paper/2410.16377/) (2024)
- [When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale](https://mlsanity.com/paper/2309.04564/) (2023)

## Built on by (most cited first, 29 of 29)

- [Rethinking Reflection in Pre-Training](https://mlsanity.com/paper/2504.04022/) (2025)
- [Weight Ensembling Improves Reasoning in Language Models](https://mlsanity.com/paper/2504.10478/) (2025)
- [Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning](https://mlsanity.com/paper/2511.19942/) (2025)
- [Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation](https://mlsanity.com/paper/2510.04265/) (2025)
- [F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare](https://mlsanity.com/paper/2602.06717/) (2026)
- [Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking](https://mlsanity.com/paper/2510.26122/) (2025)
- [Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning](https://mlsanity.com/paper/2506.04611/) (2025)
- [Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning](https://mlsanity.com/paper/2512.16911/) (2025)
- [Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling](https://mlsanity.com/paper/2509.01649/) (2025)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model](https://mlsanity.com/paper/2601.03764/) (2026)
- [A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search](https://mlsanity.com/paper/2507.00004/) (2025)
- [BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents](https://mlsanity.com/paper/2510.23458/) (2025)
- [Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling](https://mlsanity.com/paper/2512.19905/) (2025)
- [Do Instruction-Tuned Models Always Perform Better Than Base Models? Evidence from Math and Domain-Shifted Benchmarks](https://mlsanity.com/paper/2601.13244/) (2026)
- [TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning](https://mlsanity.com/paper/2605.12236/) (2026)
- [TailSFT: Filtered Fine-Tuning Improves Post-Training Performance](https://mlsanity.com/paper/2608.25756/) (2026)
- [Computational Arbitrage in AI Model Markets](https://mlsanity.com/paper/2603.22404/) (2026)
- [Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling](https://mlsanity.com/paper/2601.08777/) (2026)
- [Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement](https://mlsanity.com/paper/2604.22110/) (2026)
- [IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs](https://mlsanity.com/paper/2607.01431/) (2026)
- [Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation](https://mlsanity.com/paper/2607.14256/) (2026)
- [Towards Valid B-Rep Generation: Training-Free Wireframe Anomaly Detection and Repair](https://mlsanity.com/paper/2608.04955/) (2026)
- [Do Quantum AIs Dream in Paths? Path-Integral Slow Thinking through Grover Interference](https://mlsanity.com/paper/2609.05842/) (2026)
- [Efficient Reasoning Exploration via State-Conditioned Latent Steering with Progress Guidance](https://mlsanity.com/paper/2609.24066/) (2026)
- [Direct Optimization of Generators for Search in Automated Theorem Proving](https://mlsanity.com/paper/2609.25575/) (2026)
- [Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations](https://mlsanity.com/paper/2609.30572/) (2026)
- [SoFT: Soft Targets for Generalizable LLM Fine-Tuning](https://mlsanity.com/paper/2609.32493/) (2026)
- [Train for Many, Update with One: Improving Multi-Sample Reasoning via Fenchel-Bregman Posterior Learning](https://mlsanity.com/paper/2610.17735/) (2026)

## Abstract (arXiv)

Recent progress in large language models (LLMs) highlights the power of scaling test-time compute to achieve strong performance on complex tasks, such as mathematical reasoning and code generation. This raises a critical question: how should model training be modified to optimize performance under a subsequent test-time compute strategy and budget? To explore this, we focus on pass@N, a simple test-time strategy that searches for a correct answer in N independent samples. We show, surprisingly, that training with cross-entropy (CE) loss can be it misaligned with pass@N in that pass@N accuracy it decreases with longer training. We explain the origins of this misalignment in terms of model overconfidence induced by CE, and experimentally verify our prediction of overconfidence as an impediment to scaling test-time compute via pass@N. Furthermore we suggest a principled, modified training loss that is better aligned to pass@N by limiting model confidence and rescuing pass@N test performance. Our algorithm demonstrates improved mathematical reasoning on MATH and MiniF2F benchmarks under several scenarios: (1) providing answers to math questions; and (2) proving theorems by searching over proof trees of varying shapes. Overall our work underscores the importance of co-designing two traditionally separate phases of LLM development: training-time protocols and test-time search and reasoning strategies.