mlsanityData to 7 Oct 2026

OckBench: Measuring the Efficiency of LLM Reasoning

arXiv 2511.05722 ↗First posted 7 Nov 2025cs.CL cs.AIcited by 10

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) such as GPT-5 and Gemini 3 have pushed the frontier of automated reasoning and code generation. Yet current benchmarks emphasize accuracy and output quality, neglecting a critical dimension: efficiency of token usage. The token efficiency is highly variable in practical. Models solving the same problem with similar accuracy can exhibit up to a 5.0× difference in token length, leading to massive gap of model reasoning ability. Such variance exposes significant redundancy, highlighting the critical need for a standardized benchmark to quantify the gap of token efficiency. Thus, we introduce OckBench, the first benchmark that jointly measures accuracy and token efficiency across reasoning and coding tasks. Our evaluation reveals that token efficiency remains largely unoptimized across current models, significantly inflating serving costs and latency. These findings provide a concrete roadmap for the community to optimize the latent reasoning ability, token efficiency. Ultimately, we argue for an evaluation paradigm shift: tokens must not be multiplied beyond necessity. Our benchmarks are available at https://ockbench.github.io/.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 34 of its 40 references that are in the map. First posted on arXiv 7 Nov 2025.

Builds on

34 of 40 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025DART: Difficulty-Adaptive Reasoning Truncation for Efficient Large Language Models3 Nov
  2. 2025Revisiting Model Interpolation for Efficient Reasoning13 Oct
  3. 2025OptimalThinkingBench: Evaluating Over and Underthinking in LLMs18 Aug
  4. 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models28 May
  5. 2025The Overthinker's DIET: Cutting Token Calories with DIfficulty-AwarE Training25 May
  6. 2025Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models22 May
  7. 2025THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Models17 Apr
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  9. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies10 Jun
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May

Other references · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2025TTRL: Test-Time Reinforcement Learningcited by 253
12 more
  1. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  2. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  3. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  4. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 85
  5. 2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 73
  6. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  7. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 36
  8. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
  9. 2025Walk Before You Run! Concise LLM Reasoning via Reinforcement Learningcited by 15
  10. 2025Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Modelscited by 11
  11. 2025Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMscited by 9
  12. 2024UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughtscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026El Agente Gr'afico: A Semantic Execution Runtime for Scientific Agentscited by 4
  2. 2026OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agentscited by 4
  3. 2026CurveShift: Is Agent Progress Scalar? Separating Level from Shapecited by 2
  4. 2026Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMscited by 1
  5. 2026GraphRAG for Engineering Diagrams: ChatP&ID Enables LLM Interaction with P&IDscited by 1
  6. 2026Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessingcited by 1
  7. 2026Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curationcited by 0
  8. 2026When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficultycited by 0
  9. 2026The Reasoning Tax: Token Economics of LLM Reasoning Across Task Types and Deployment Contextscited by 0
  10. 2026Reputation as Community Memory for the Agentic Webcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
  2. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  3. 2026On the Token Value Inequality in Efficient Reasoning
  4. 2025OJBench: A Competition Level Code Benchmark For Large Language Models
  5. 2025EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Models
  6. 2026Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
  7. 2024Inference Scaling vs Reasoning: An Empirical Analysis of Compute-Optimal LLM Problem-Solving
  8. 2024Extending Token Computation for LLM Reasoning