Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generated, not how they are executed. The same candidate budget can be produced in one batched generation call or split across several sequential calls with smaller batch sizes. We first study the effect of increasing N on reasoning accuracy using Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts. As expected, increasing N from 1 to 8 improves accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. However, accuracy alone does not show the systems cost of using a larger candidate budget. We therefore fix N = 8 and compare four generation schedules: 1x8, 2x4, 4x2, and 8x1, where axb denotes a generation calls with b candidates per call. We measure latency, throughput, GPU-hours, and gross GPU-device energy while keeping the total candidate count fixed. On A100 GPUs, eight serial calls use 4.64-4.86x as much gross GPU-device energy and have 5.77-6.12x the P95 latency of one batched call with eight candidates. The same pattern appears across three independently scheduled A100 nodes per model and in short-output SciQ/V100 experiments. These results show that candidate count alone is not enough to describe the systems cost of multi-candidate test-time scaling. When candidates are independent and memory allows it, fewer generation calls with larger batch sizes are more efficient. Evaluations should therefore report not only candidate count and accuracy, but also generation schedule and GPU-level systems metrics.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 31 of its 38 references that are in the map. First posted on arXiv 16 Sep 2026.
Builds on
31 of 38 references.
Closest ancestors · 11
Same direction, newest first
- 2026Interpretable Adaptive Sampling for LLM Test-Time Scaling4 Aug
- 2026Reliable Chain-of-Thought via Prefix Consistency8 May
- 2026Reliability-Aware Adaptive Self-Consistency for Efficient Sampling in LLM Reasoning6 Jan
- 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models4 Jun
- 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?31 Mar
- 2025Scalable Best-of-N Selection for Large Language Models via Self-Certainty25 Feb
- 2025s1: Simple test-time scaling31 Jan
- 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning24 Aug
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
- 2023Universal Self-Consistency for Large Language Model Generation29 Nov
- 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs19 May
Other references · 20
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
- 2017Crowdsourcing Multiple Choice Science Questionscited by 765
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
- 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
- 2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 76
- 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
- 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
8 more
- 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 55
- 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
- 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 28
- 2024PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Servicescited by 23
- 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 19
- 2025Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulationscited by 12
- 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architecturescited by 5
- 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Usecited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025The Art of Scaling Test-Time Compute for Large Language Models
- 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute
- 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
- 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
- 2025Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory
- 2025Scaling Test-time Compute for LLM Agents
- 2025A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning
- 2025Latency and Token-Aware Test-Time Compute