mlsanityData to 7 Oct 2026

Variational Proximal Policy Optimization

arXiv 2606.08032 ↗First posted 6 Jun 2026stat.ML cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback via Proximal Policy Optimization often suffers from policy mode collapse, brittle exploration loops, and distribution drift. This paper introduces Variational Proximal Policy Optimization ((textscVP2textscO)), a particle-based variational inference framework that maps policy optimization to Stein Variational Gradient Descent within a Mixture-of-Experts architecture. By leveraging functional kernels over localized expert prototypes alongside an expert orthogonalization loss, (textscVP2textscO) introduces a geometry-based proximal-control mechanism that can reduce reliance on fixed clipping or KL schedules. Our results on a 33B/4B sparse Mixture-of-Experts model show several improvements across complex reasoning benchmarks, establishing a (+179) ELO gain on Codeforces and a (32%) reduction in token count on AIME mathematical reasoning tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 24 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

17 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  12. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
3 more
  1. 2017Stein Variational Policy Gradientcited by 42
  2. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  3. 2025Advancing Expert Specialization for Better MoEcited by 37

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  2. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
  3. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning
  4. 2026Ratio-Variance Regularized Policy Optimization
  5. 2025Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization
  6. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
  7. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  8. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning