# Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accurately arXiv: https://arxiv.org/abs/2505.13326 ยท HTML: https://mlsanity.com/paper/2505.13326/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. - Builds on 26 of its 33 references that are in the map. - First posted on arXiv 19 May 2025. ## Placed in - LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/) ## Builds on (most cited first, 26 of 26) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024) - [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025) - [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025) - [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023) - [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025) - [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023) - [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025) - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025) - [Dynamic Early Exit in Reasoning Models](https://mlsanity.com/paper/2504.15895/) (2025) - [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023) - [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023) - [Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs](https://mlsanity.com/paper/2505.00127/) (2025) - [Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs](https://mlsanity.com/paper/2305.11860/) (2023) - [ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning](https://mlsanity.com/paper/2504.21370/) (2025) - [Rewarding Chatbots for Real-World Engagement with Millions of Users](https://mlsanity.com/paper/2303.06135/) (2023) - [CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Model](https://mlsanity.com/paper/2502.11169/) (2025) - [Towards Effective Disambiguation for Machine Translation with Large Language Models](https://mlsanity.com/paper/2309.11668/) (2023) ## Built on by (most cited first, 6 of 6) - [Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2507.09662/) (2025) - [BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2602.10729/) (2026) - [Understanding the Role of Training Data in Test-Time Scaling](https://mlsanity.com/paper/2510.03605/) (2025) - [OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration](https://mlsanity.com/paper/2602.12151/) (2026) - [EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams](https://mlsanity.com/paper/2605.07299/) (2026) - [Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics](https://mlsanity.com/paper/2604.07144/) (2026) ## Abstract (arXiv) Recent advances in test-time scaling suggest that Large Language Models (LLMs) can gain better capabilities by generating Chain-of-Thought reasoning (analogous to human thinking) to respond a given request, and meanwhile exploring more reasoning branches (i.e., generating multiple responses and ensembling them) can improve the final output quality. However, when incorporating the two scaling dimensions, we find that the system efficiency is dampened significantly for two reasons. Firstly, the time cost to generate the final output increases substantially as many reasoning branches would be trapped in the over-thinking dilemma, producing excessively long responses. Secondly, generating multiple reasoning branches for each request increases memory consumption, which is unsuitable for LLM serving since we can only batch a limited number of requests to process simultaneously. To address this, we present SART, a serving framework for efficient and accurate LLM reasoning. The essential idea is to manage the thinking to be short and right, rather than long. For one thing, we devise a redundant sampling with early stopping approach based on empirical observations and theoretic analysis, which increases the likelihood of obtaining short-thinking responses when sampling reasoning branches. For another, we propose to dynamically prune low-quality branches so that only right-thinking branches are maintained, reducing the memory consumption and allowing us to batch more requests. Experimental results demonstrate that SART not only improves the accuracy of LLM reasoning but also enhances the serving efficiency, outperforming existing methods by up to 28.2 times and on average 15.7 times in terms of efficiency when achieving the same level of accuracy.