# Latency and Token-Aware Test-Time Compute

arXiv: https://arxiv.org/abs/2509.09864 · HTML: https://mlsanity.com/paper/2509.09864/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 13 of 13)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [When More is Less: Understanding Chain-of-Thought Length in LLMs](https://mlsanity.com/paper/2502.07266/) (2025)
- [Learning How Hard to Think: Input-Adaptive Allocation of LM Computation](https://mlsanity.com/paper/2410.04707/) (2024)
- [BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute](https://mlsanity.com/paper/2506.22716/) (2025)
- [Causal LLM Routing: End-to-End Regret Minimization from Observational Data](https://mlsanity.com/paper/2505.16037/) (2025)
- [Scaling LLM Inference with Optimized Sample Compute Allocation](https://mlsanity.com/paper/2410.22480/) (2024)
- [AdaBoN: Adaptive Best-of-N Alignment](https://mlsanity.com/paper/2505.12050/) (2025)
- [Optimal Policy Minimum Bayesian Risk](https://mlsanity.com/paper/2505.17242/) (2025)

## Built on by (most cited first, 8 of 8)

- [A Survey on Parallel Reasoning](https://mlsanity.com/paper/2510.12164/) (2025)
- [Seer Self-Consistency: Advance Budget Estimation for Adaptive Test-Time Scaling](https://mlsanity.com/paper/2511.09345/) (2025)
- [On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows](https://mlsanity.com/paper/2605.06110/) (2026)
- [Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systems](https://mlsanity.com/paper/2601.19311/) (2026)
- [Decision-Centric Design for LLM Systems](https://mlsanity.com/paper/2604.00414/) (2026)
- [CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing](https://mlsanity.com/paper/2608.07424/) (2026)
- [Intelligence Under Time Constraints: Rethinking Test-Time Compute](https://mlsanity.com/paper/2609.14995/) (2026)
- [From Pareto to Preference: Personalized Test-Time Scaling via Amortized Agentic Policy Discovery](https://mlsanity.com/paper/2610.09684/) (2026)

## Abstract (arXiv)

Inference-time scaling has emerged as a powerful way to improve large language model (LLM) performance by generating multiple candidate responses and selecting among them. However, existing work on dynamic allocation for test-time compute typically considers only parallel generation methods such as best-of-N, overlooking incremental decoding methods like beam search, and has largely ignored latency, focusing only on token usage. We formulate inference-time scaling as a problem of dynamic compute allocation and method selection, where the system must decide which strategy to apply and how much compute to allocate on a per-query basis. Our framework explicitly incorporates both token cost and wall-clock latency, the latter being critical for user experience and particularly for agentic workflows where models must issue multiple queries efficiently. Experiments on reasoning benchmarks show that our approach consistently outperforms static strategies, achieving favorable accuracy-cost trade-offs while remaining practical for deployment.