mlsanityData to 14 Oct 2026

Kinetics: Rethinking Test-Time Scaling Laws

arXiv 2506.05333 ↗First posted 5 Jun 2025cs.LG cs.CLcited by 12

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

We rethink test-time scaling laws from a practical efficiency perspective, revealing that the effectiveness of smaller models is significantly overestimated. Prior work, grounded in compute-optimality, overlooks critical memory access bottlenecks introduced by inference-time strategies (e.g., Best-of-N, long CoTs). Our holistic analysis, spanning models from 0.6B to 32B parameters, reveals a new Kinetics Scaling Law that better guides resource allocation by incorporating both computation and memory access costs. Kinetics Scaling Law suggests that test-time compute is more effective when used on models above a threshold than smaller ones. A key reason is that in TTS, attention, rather than parameter count, emerges as the dominant cost factor. Motivated by this, we propose a new scaling paradigm centered on sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget. Empirically, we show that sparse attention models consistently outperform dense counterparts, achieving over 60 points gains in low-cost regimes and over 5 points gains in high-cost regimes for problem-solving accuracy on AIME, encompassing evaluations on state-of-the-art MoEs. These results suggest that sparse attention is essential and increasingly important with more computing invested, for realizing the full potential of test-time scaling where, unlike training, accuracy has yet to saturate as a function of computation, and continues to improve through increased generation. The code is available at https://github.com/Infini-AI-Lab/Kinetics.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 72 of its 110 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

72 of 110 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Parallel Scaling Law for Language Models15 May
  2. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models14 Apr
  3. 2025Reasoning Models Can Be Effective Without Thinking14 Apr
  4. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners27 Feb
  5. 2025RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning16 Feb
  6. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuning13 Feb
  7. 2025LLM Pretraining with Continuous Concepts12 Feb
  8. 2025s1: Simple test-time scaling31 Jan
  9. 2024Efficiently Scaling LLM Reasoning with Certaindex30 Dec
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  11. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models1 Aug
  12. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training29 Sep
1 more
  1. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation28 Jul

Other references · 59

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2020Longformer: The Long-Document Transformercited by 3,413
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2024Mixtral of Expertscited by 1,823
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2020Rethinking Attention with Performerscited by 1,601
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  11. 2024OpenAI o1 System Cardcited by 1,244
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  13. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  14. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  15. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  16. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  17. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  18. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  19. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  20. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  21. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  22. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  23. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  24. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  25. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  26. 2021Accelerating Sparse Deep Neural Networkscited by 223
  27. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  28. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  29. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  30. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  31. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  32. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  33. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  34. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  35. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  36. 2024Scaling Laws for Precisioncited by 95
  37. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
  38. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipescited by 7
  2. 2026Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learningcited by 5
  3. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 5
  4. 2025Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Modelscited by 4
  5. 2026Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scalingcited by 3
  6. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  7. 2026Unified Neural Scaling Lawscited by 2
  8. 2026Prefix Sliding for efficient test-time scalingcited by 2
  9. 20253D Optimization for AI Inference Scaling: Balancing Accuracy, Cost, and Latencycited by 1
  10. 2026Interpretable Adaptive Sampling for LLM Test-Time Scalingcited by 1
  11. 2025BlindSight: Harnessing Sparsity for Efficient Vision-Language Modelscited by 0
  12. 2026How Much Can Language Models Gain from Test-Time Computation?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Test-Time Scaling Makes Overtraining Compute-Optimal
  2. 2025Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling
  3. 2025Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Models
  4. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
  5. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
  6. 2026Towards Better Exploration in Sequential Test-Time Scaling
  7. 2025ETS: Efficient Tree Search for Inference-Time Scaling
  8. 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck