# Flexible Empowerment at Reasoning with Extended Best-of-N Sampling

arXiv: https://arxiv.org/abs/2604.15614 · HTML: https://mlsanity.com/paper/2604.15614/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 30 of 31)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [Sparse Sequence-to-Sequence Models](https://mlsanity.com/paper/1905.05702/) (2019)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Boltzmann Exploration Done Right](https://mlsanity.com/paper/1705.10257/) (2017)
- [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023)
- [Evaluation of Best-of-N Sampling Strategies for Language Model Alignment](https://mlsanity.com/paper/2502.12668/) (2025)
- [Continual Learning as Computationally Constrained Reinforcement Learning](https://mlsanity.com/paper/2307.04345/) (2023)
- [Soft Best-of-n Sampling for Model Alignment](https://mlsanity.com/paper/2505.03156/) (2025)
- [A Max-Min Entropy Framework for Reinforcement Learning](https://mlsanity.com/paper/2106.10517/) (2021)
- [INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL](https://mlsanity.com/paper/2204.08585/) (2022)
- [Latent Exploration for Reinforcement Learning](https://mlsanity.com/paper/2305.20065/) (2023)
- [Squareplus: A Softplus-Like Algebraic Rectifier](https://mlsanity.com/paper/2112.11687/) (2021)
- [R'enyi State Entropy for Exploration Acceleration in Reinforcement Learning](https://mlsanity.com/paper/2203.04297/) (2022)
- [AdaSplash: Adaptive Sparse Flash Attention](https://mlsanity.com/paper/2502.12082/) (2025)
- [AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization](https://mlsanity.com/paper/2201.06714/) (2022)
- [Consolidated Adaptive T-soft Update for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.12504/) (2022)
- [Reward Bonuses with Gain Scheduling Inspired by Iterative Deepening Search](https://mlsanity.com/paper/2212.10765/) (2022)
- [Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2405.17243/) (2024)

## Built on by (most cited first, 3 of 3)

- [Redesigning Regularization for Effective Policy Smoothing](https://mlsanity.com/paper/2606.13169/) (2026)
- [Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs](https://mlsanity.com/paper/2606.17735/) (2026)
- [ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignment](https://mlsanity.com/paper/2609.21899/) (2026)

## Abstract (arXiv)

This paper proposes a novel method that incorporates empowerment when reasoning actions in reinforcement learning (RL), thereby achieving the flexibility of exploration-exploitation dilemma (EED). In previous methods, empowerment for promoting exploration has been provided as a bonus term to the task-specific reward function as an intrinsically-motivated RL. However, this approach introduces a delay until the policy that accounts for empowerment is learned, making it difficult to adjust the emphasis on exploration as needed. On the other hand, a trick devised for fine-tuning recent foundation models at reasoning, so-called best-of-N (BoN) sampling, allows for the implicit acquisition of modified policies without explicitly learning them. It is expected that applying this trick to exploration-promoting terms, such as empowerment, will enable more flexible adjustment of EED. Therefore, this paper investigates BoN sampling for empowerment. Furthermore, to adjust the degree of policy modification in a generalizable manner while maintaining computational cost, this paper proposes a novel BoN sampling method extended by Tsalis statistics. Through toy problems, the proposed method's cability to balance EED is verified. In addition, it is demonstrated that the proposed method improves RL performance to solve complex locomotion tasks.