mlsanityData to 7 Oct 2026

Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization

arXiv 2501.17974 ↗First posted 29 Jan 2025cs.AIcited by 29

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Solving mathematics problems has been an intriguing capability of large language models, and many efforts have been made to improve reasoning by extending reasoning length, such as through self-correction and extensive long chain-of-thoughts. While promising in problem-solving, advanced long reasoning chain models exhibit an undesired single-modal behavior, where trivial questions require unnecessarily tedious long chains of thought. In this work, we propose a way to allow models to be aware of inference budgets by formulating it as utility maximization with respect to an inference budget constraint, hence naming our algorithm Inference Budget-Constrained Policy Optimization (IBPO). In a nutshell, models fine-tuned through IBPO learn to ``understand'' the difficulty of queries and allocate inference budgets to harder ones. With different inference budgets, our best models are able to have a 4.14% and 5.74% absolute improvement (8.08% and 11.2% relative improvement) on MATH500 using 2.16x and 4.32x inference budgets respectively, relative to LLaMA3.1 8B Instruct. These improvements are approximately 2x those of self-consistency under the same budgets.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 57 of its 83 references that are in the map. First posted on arXiv 29 Jan 2025.

Builds on

57 of 83 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  2. 2024S^3cMath: Spontaneous Step-level Self-correction Makes Large Language Models Better Mathematical Reasoners3 Sep
  3. 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning24 Aug
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  5. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  6. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs19 May
  7. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  8. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  11. 2023Let's Verify Step by Stepcited by 3,598
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
37 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2019Introduction to Online Convex Optimizationcited by 793
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
  13. 2019Introduction to Multi-Armed Banditscited by 475
  14. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  15. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  16. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  17. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  18. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  19. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  20. 2020Projection-Based Constrained Policy Optimizationcited by 195
  21. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 168
  22. 2013Bandits with Knapsackscited by 163
  23. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  24. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  25. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  26. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  27. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  28. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 57
  29. 2023Let Models Speak Ciphers: Multiagent Debate through Embeddingscited by 49
  30. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  31. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  32. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39
  33. 2023Goodhart's Law in Reinforcement Learningcited by 37
  34. 2023B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesiscited by 16
  35. 2016Guided Policy Search as Approximate Mirror Descentcited by 14
  36. 2015Bandit Convex Optimization: sqrt{T} Regret in One Dimensioncited by 10
  37. 2021A Lyapunov-Based Methodology for Constrained Optimization with Bandit Feedbackcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 22; 2026: 7.

29 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  4. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  5. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  6. 2025AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgetingcited by 50
  7. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  8. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  9. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokenscited by 27
  10. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  11. 2025Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoningcited by 21
  12. 2025Know What You Don't Know: Uncertainty Calibration of Process Reward Modelscited by 19
17 more
  1. 2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
  2. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  3. 2025From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Modelscited by 11
  4. 2025Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theorycited by 10
  5. 2025C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoningcited by 8
  6. 2025DynScaling: Efficient Verifier-free Inference Scaling via Dynamic and Integrated Samplingcited by 7
  7. 2025DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chainscited by 7
  8. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 7
  9. 2026Ares: Adaptive Reasoning Effort Selection for Efficient LLM Agentscited by 7
  10. 2025RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMscited by 5
  11. 2025Adaptive Rectification Sampling for Test-Time Compute Scalingcited by 3
  12. 2026UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scalingcited by 1
  13. 2025EasyMath: A 0-shot Math Benchmark for SLMscited by 0
  14. 2026HyPER: Bridging Exploration and Exploitation for Scalable LLM Reasoning with Hypothesis Path Expansion and Reductioncited by 0
  15. 2026Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learningcited by 0
  16. 2026Post-Reasoning: Improving the Performance of Non-Thinking Models at No Costcited by 0
  17. 2026LIMBO: Lifelong Inference-Time Memory and Budget Optimization for LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation
  2. 2026Nice Fold or Hero Call: Learning Budget-Efficient Thinking for Adaptive Reasoning
  3. 2025Optimizing Length Compression in Large Reasoning Models
  4. 2026ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement
  5. 2026Efficient Reasoning via Constrained Optimization in Latent Space
  6. 2026Knowing When Thinking Is Not Enough: Teaching Small Reasoning Models to Reason Beyond Their Parametric Knowledge
  7. 2026Stabilizing Efficient Reasoning with Step-Level Advantage Selection
  8. 2025ORION: Teaching Language Models to Reason Efficiently in the Language of Thought