mlsanityData to 7 Oct 2026

Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs

arXiv 2605.04065 ↗First posted 11 Apr 2026cs.CL cs.ET cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Unsupervised reinforcement learning (RL) has emerged as a promising paradigm for enabling self-improvement in large language models (LLMs). However, existing unsupervised RL-based methods often lack the capacity to adapt to the model's evolving reasoning capabilities during training. Therefore, these methods can misdirect policy optimization in the absence of ground-truth supervision. To address this issue, we introduce FREIA, a novel RL-based algorithm built on two key innovations: (1) Free Energy-Driven Reward (FER) adapts rewards to balance consensus and exploration based on the Free Energy Principle. (2) Adaptive Advantage Shaping (AAS) adaptively adjusts learning signals based on the statistical characteristics of sampled rewards. Empirical evaluations on nine datasets across three reasoning tasks showcase that FREIA outperforms other unsupervised RL-based baselines. Notably, in mathematical reasoning tasks, FREIA surpasses other methods by an average of 0.5 to 3.5 points in Pass@1 using the DeepSeek-R1-Distill-Qwen-1.5B model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 33 references that are in the map. First posted on arXiv 11 Apr 2026.

Builds on

20 of 33 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward3 Oct
  2. 2025Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedback17 Aug
  3. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning20 Jun
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  5. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models5 Jun
  6. 2025Maximizing Confidence Alone Improves Reasoning28 May
  7. 2025Learning to Reason without External Rewards26 May
  8. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning23 May
  9. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  10. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 9

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2024Self-Consistency Preference Optimizationcited by 50
  8. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  9. 2025GeometryZero: Advancing Geometry Solving via Group Contrastive Policy Optimizationcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  2. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
  3. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  4. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  5. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  6. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2025Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models