mlsanityData to 14 Oct 2026

A Critical Look at Power Sampling when Test Time Sharpening LLMs

arXiv 2610.13653 ↗First posted 9 Oct 2026stat.ML cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution πα propto πbaseα via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to πα under realistic compute budgets, and that their gains are largely recreated via simple Best-of-N with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating πα, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match πα, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting πα, but in the simpler goal of extracting sequences the base model already deems likely.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 22 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

21 of 22 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening29 Jan
  2. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Think16 Oct
  3. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  4. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  5. 2025Learning to Reason without External Rewards26 May
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar

Other references · 13

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2024Self-Rewarding Language Modelscited by 681
  8. 2022Large Language Models Can Self-Improvecited by 664
  9. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  10. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  11. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  12. 2026Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoningcited by 21
1 more
  1. 2025Bourbaki: Self-Generated and Goal-Conditioned MDPs for Theorem Provingcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
  2. 2026Explore Broadly, Reason Sharply: Push Small Models toward the Frontier via Sampling
  3. 2026Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewards
  4. 2026Decoupled Marginal Sharpening for Training-Free Inference-Time Scaling
  5. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimal
  6. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  7. 2026When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewards
  8. 2026More Correct Mass, Worse Answers: Why Power Sampling Can Fail and How to Fix It