mlsanityData to 7 Oct 2026

Rational Metareasoning for Large Language Models

arXiv 2410.05563 ↗First posted 7 Oct 2024cs.CL cs.AI cs.LGcited by 20

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Being prompted to engage in reasoning has emerged as a core technique for using large language models (LLMs), deploying additional inference-time compute to improve task performance. However, as LLMs increase in both size and adoption, inference costs are correspondingly becoming increasingly burdensome. How, then, might we optimize reasoning's cost-performance tradeoff? This work introduces a novel approach based on computational models of metareasoning used in cognitive science, training LLMs to selectively use intermediate reasoning steps only when necessary. We first develop a reward function that incorporates the Value of Computation by penalizing unnecessary reasoning, then use this reward function with Expert Iteration to train the LLM. Compared to few-shot chain-of-thought prompting and STaR, our method significantly reduces inference costs (20-37% fewer tokens generated across three models) while maintaining task performance across diverse datasets.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 44 of its 69 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

44 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  2. 2023Think before you speak: Training Language Models With Pause Tokens3 Oct
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  4. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  5. 2022Large Language Models Can Self-Improve20 Oct
  6. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2024Mixtral of Expertscited by 1,785
26 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  4. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  5. 2018Universal Transformerscited by 613
  6. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  8. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  9. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  10. 2022Confident Adaptive Language Modelingcited by 297
  11. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  13. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  14. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  15. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  16. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  17. 2023Efficient Large Language Models: A Surveycited by 183
  18. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  19. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 157
  20. 2021PonderNet: Learning to Pondercited by 156
  21. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  22. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  23. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  24. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103
  25. 2023CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inferencecited by 27
  26. 2024Synergy-of-Thoughts: Eliciting Efficient Reasoning in Hybrid Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 10.

20 papers.

Sort:

By area:

  1. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  2. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  3. 2025Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoningcited by 16
  4. 2025Levels of Analysis for Large Language Modelscited by 9
  5. 2025Are Large Language Models Sensitive to the Motives Behind Communication?cited by 7
  6. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  7. 2025Monitor-Generate-Verify (MGV): Formalising Metacognitive Theory for Language Model Reasoningcited by 4
  8. 2026Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agentscited by 4
  9. 2025Verifying Meta-Awareness via Predictive Rewards in Reasoning Modelscited by 3
  10. 2025Searching Meta Reasoning Skeleton to Guide LLM Reasoningcited by 3
  11. 2026ATLAS: Agentic Test-time Learning-to-Allocate Scalingcited by 3
  12. 2026Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocationcited by 2
8 more
  1. 2025Metareasoning constraints couple narratives, affect and cognitioncited by 1
  2. 2026When to Plan: Learning to Select Between Reactive Control and Deliberative Planningcited by 1
  3. 2026Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questionscited by 1
  4. 2026When does a scaling result justify a different allocation? A critical review of resource-allocation evidence for AI systemscited by 0
  5. 2026Intelligence Under Time Constraints: Rethinking Test-Time Computecited by 0
  6. 2026PrimeScientist: Strategic Allocation of Research Effort in Autonomous Researchcited by 0
  7. 2026When Should an In-Context Learner Expand Its Hypothesis Space?cited by 0
  8. 2026LEVER: Adaptive Cost-Aware Proof Search Over AND/OR Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Cognitive Demand Steering for Adaptive Meta-Reasoning in Large Language Models
  2. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
  3. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
  4. 2026Enhancing reasoning accuracy in large language models during inference time
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2025Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models
  7. 2025LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?
  8. 2025SplitReason: Learning To Offload Reasoning