# PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching

arXiv: https://arxiv.org/abs/2603.18363 · HTML: https://mlsanity.com/paper/2603.18363/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 46 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Builds on (most cited first, 30 of 38)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Trajectory balance: Improved credit assignment in GFlowNets](https://mlsanity.com/paper/2201.13259/) (2022)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [Spurious Rewards: Rethinking Training Signals in RLVR](https://mlsanity.com/paper/2506.10947/) (2025)
- [Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization](https://mlsanity.com/paper/2504.05812/) (2025)
- [Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity](https://mlsanity.com/paper/2510.01171/) (2025)
- [Self-Improvement in Language Models: The Sharpening Mechanism](https://mlsanity.com/paper/2412.01951/) (2024)
- [Maximizing Confidence Alone Improves Reasoning](https://mlsanity.com/paper/2505.22660/) (2025)
- [Reasoning with Sampling: Your Base Model is Smarter Than You Think](https://mlsanity.com/paper/2510.14901/) (2025)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [Can Large Reasoning Models Self-Train?](https://mlsanity.com/paper/2505.21444/) (2025)
- [Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models](https://mlsanity.com/paper/2506.06395/) (2025)
- [Diverse Preference Optimization](https://mlsanity.com/paper/2501.18101/) (2025)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [FlowRL: Matching Reward Distributions for LLM Reasoning](https://mlsanity.com/paper/2509.15207/) (2025)

## Built on by (most cited first, 2 of 2)

- [Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning](https://mlsanity.com/paper/2608.03545/) (2026)
- [One Proposal for Every Margin: Zero-Shot Amortized Sequential Importance Sampling for Binary Matrices](https://mlsanity.com/paper/2609.35514/) (2026)

## Abstract (arXiv)

Unsupervised Reinforcement Learning from Internal Feedback (RLIF) has emerged as a promising paradigm for eliciting the latent capabilities of Large Language Models (LLMs) without external supervision. However, current methods rely on heuristic intrinsic rewards, which often lack a well-defined theoretical optimization target and are prone to degenerative biases. In this work, we introduce PowerFlow, a principled framework that reformulates unsupervised fine-tuning as a distribution matching problem. By casting GFlowNet as an amortized variational sampler for unnormalized densities, we propose a length-aware Trajectory-Balance objective that explicitly neutralizes the structural length biases inherent in autoregressive generation. By targeting α-power distributions, PowerFlow enables the directional elicitation of the dual nature of LLMs: sharpening the distribution (α > 1) to intensify logical reasoning, or flattening it (α < 1) to unlock expressive creativity. Extensive experiments demonstrate that PowerFlow consistently outperforms existing RLIF methods, matching or even exceeding supervised GRPO. Furthermore, by mitigating over-sharpening in aligned models, our approach achieves simultaneous gains in diversity and quality, shifting the Pareto frontier in creative tasks.