mlsanityData to 7 Oct 2026

PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore

arXiv 2511.08241 ↗First posted 11 Nov 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Exploration in reinforcement learning remains a critical challenge, as naive entropy maximization often results in high variance and inefficient policy updates. We introduce PrefPoE, a novel textitPreference-Product-of-Experts framework that performs intelligent, advantage-guided exploration via the first principled application of product-of-experts (PoE) fusion for single-task exploration-exploitation balancing. By training a preference network to concentrate probability mass on high-advantage actions and fusing it with the main policy through PoE, PrefPoE creates a soft trust region that stabilizes policy updates while maintaining targeted exploration. Across diverse control tasks spanning both continuous and discrete action spaces, PrefPoE demonstrates consistent improvements: +321% on HalfCheetah-v4 (1276~rightarrow~5375), +69% on Ant-v4, +276% on LunarLander-v2, with consistently enhanced training stability and sample efficiency. Unlike standard PPO, which suffers from entropy collapse, PrefPoE sustains adaptive exploration through its unique dynamics, thereby preventing premature convergence and enabling superior performance. Our results establish that learning textitwhere to explore through advantage-guided preferences is as crucial as learning how to act, offering a general framework for enhancing policy gradient methods across the full spectrum of reinforcement learning domains. Code and pretrained models are available in supplementary materials.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

21 of 34 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  8. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  9. 2019Challenges of Real-World Reinforcement Learningcited by 282
  10. 2017Parameter Space Noise for Explorationcited by 158
  11. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  12. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
9 more
  1. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  2. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  3. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  4. 2017Divide-and-Conquer Reinforcement Learningcited by 46
  5. 2020Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Controlcited by 44
  6. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
  7. 2023Optimal Exploration for Model-Based RL in Nonlinear Systemscited by 17
  8. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  9. 2023Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RLcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Targeted Exploration via Unified Entropy Control for Reinforcement Learning
  2. 2024Preference-Guided Reinforcement Learning for Efficient Exploration
  3. 2026Expert-Space Exploration in MoE Reinforcement Learning
  4. 2026Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning
  5. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning
  6. 2025PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization
  7. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  8. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning