# Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling arXiv: https://arxiv.org/abs/2508.01969 · HTML: https://mlsanity.com/paper/2508.01969/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. - Builds on 28 of its 40 references that are in the map. - First posted on arXiv 4 Aug 2025. ## Placed in - LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/) - LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/) ## Builds on (most cited first, 28 of 28) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023) - [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024) - [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022) - [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023) - [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023) - [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024) - [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018) - [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024) - [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://mlsanity.com/paper/2205.09712/) (2022) - [Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training](https://mlsanity.com/paper/2309.17179/) (2023) - [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025) - [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024) - [V-STaR: Training Verifiers for Self-Taught Reasoners](https://mlsanity.com/paper/2402.06457/) (2024) - [Generative Reward Models](https://mlsanity.com/paper/2410.12832/) (2024) - [Fast Best-of-N Decoding via Speculative Rejection](https://mlsanity.com/paper/2410.20290/) (2024) - [CriticBench: Benchmarking LLMs for Critique-Correct Reasoning](https://mlsanity.com/paper/2402.14809/) (2024) - [Prover-Verifier Games improve legibility of LLM outputs](https://mlsanity.com/paper/2407.13692/) (2024) - [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://mlsanity.com/paper/2406.16253/) (2024) - [Empowering LLMs with Logical Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2502.15652/) (2025) - [AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition](https://mlsanity.com/paper/2402.11452/) (2024) - [Critique Ability of Large Language Models](https://mlsanity.com/paper/2310.04815/) (2023) - [Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models](https://mlsanity.com/paper/2503.13551/) (2025) - [A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Length](https://mlsanity.com/paper/2407.05347/) (2024) - [VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifiers](https://mlsanity.com/paper/2410.08048/) (2024) - [Have Large Language Models Learned to Reason? A Characterization via 3-SAT Phase Transition](https://mlsanity.com/paper/2504.03930/) (2025) - [PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Search](https://mlsanity.com/paper/2504.10222/) (2025) - [Improving Math Problem Solving in Large Language Models Through Categorization and Strategy Tailoring](https://mlsanity.com/paper/2411.00042/) (2024) ## Built on by (most cited first, 4 of 4) - [Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts](https://mlsanity.com/paper/2509.21743/) (2025) - [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026) - [Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing](https://mlsanity.com/paper/2606.02218/) (2026) - [Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection](https://mlsanity.com/paper/2605.14062/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) are increasingly relied upon for solving complex reasoning tasks in domains such as mathematics, logic, and multi-step question answering. A growing line of work seeks to improve reasoning quality by scaling inference time compute particularly through Process Reward Models (PRMs), used to reward the reasoning at intermediate steps. While effective, these methods introduce substantial computational overhead, especially when generating large numbers of solutions in parallel. In this paper, we investigate whether PRMs can be used mid-generation to provide early signals that enable the rejection of suboptimal candidates before full generation of step is complete. We introduce the hypothesis that PRMs are also Partial Reward Models, meaning that the scores they assign to partially completed reasoning step are predictive of final output quality. This allows for principled early rejection based on intermediate token-level signals. We support this hypothesis both theoretically, by proving that the risk of discarding optimal beams decreases exponentially with generation length and empirically, by demonstrating a strong correlation between partial and final rewards across multiple reward models. On math reasoning benchmarks, our method achieves up to 1.4×-9× reduction in inference FLOPs without degrading final performance. These results suggest that early rejection is a powerful mechanism for improving the compute-efficiency of reasoning in LLMs.