mlsanityData to 14 Oct 2026

ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment

arXiv 2601.21484 ↗First posted 29 Jan 2026cs.LGcited by 5

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) post-training alignment for language models is effective, but also costly and unstable in practice, owing to its complicated training process. To address this, we propose a training-free inference method to sample directly from the optimal RL policy. The transition probability applied to Masked Language Modeling (MLM) consists of a reference policy model and an energy term. Based on this, our algorithm, Energy-Guided Test-Time Scaling (ETS), estimates the key energy term via online Monte Carlo, with a provable convergence rate. Moreover, to ensure practical efficiency, ETS leverages modern acceleration frameworks alongside tailored importance sampling estimators, substantially reducing inference latency while provably preserving sampling quality. Experiments on MLM (including autoregressive models and diffusion language models) across reasoning, coding, and science benchmarks show that our ETS consistently improves generation quality, validating its effectiveness and design. The code is available at https://github.com/sheriyuo/ETS.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 84 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

45 of 84 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts15 Jan
  2. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Think16 Oct
  3. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  4. 2025Can Large Reasoning Models Self-Train?27 May
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2025Large Language Diffusion Modelscited by 912
  12. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
26 more
  1. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  4. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  5. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  6. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 224
  7. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  9. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  10. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 143
  11. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  12. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  13. 2025LongCat-Image Technical Reportcited by 100
  14. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  15. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  16. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 41
  17. 2023On the Generalization of Diffusion Modelcited by 30
  18. 2025Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewardscited by 27
  19. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  20. 2025An Empirical Study of Qwen3 Quantizationcited by 21
  21. 2025SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learningcited by 9
  22. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithmscited by 8
  23. 2025Inference-Time Scaling of Diffusion Language Models via Trajectory Refinementcited by 7
  24. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 6
  25. 2024Denoising Data with Measurement Error Using a Reproducing Kernel-based Diffusion Modelcited by 3
  26. 2026TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQLcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Servicescited by 3
  2. 2026HTAM: Hierarchical Transition-Attended Memory for Operator Optimizationcited by 2
  3. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignmentcited by 2
  4. 2026SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inferencecited by 0
  5. 2026MERIT: Multimodal Evidence Reasoning over Image Tablescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Compute Aligned Training: Optimizing for Test Time Inference
  2. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  3. 2023Language Model Alignment with Elastic Reset
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  5. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
  6. 2025RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models
  7. 2025TTRL: Test-Time Reinforcement Learning
  8. 2026Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling