mlsanityData to 14 Oct 2026

Proximal Policy Optimization for Amortized Discrete Sampling

arXiv 2606.15793 ↗First posted 14 Jun 2026cs.LG cs.AI stat.MLcited by 1

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

This paper explores policy gradient algorithms for training stochastic policies to sample from structured discrete probability distributions under the Generative Flow Network (GFlowNet) framework. Building on extensive theoretical connections between GFlowNets and entropy-regularized reinforcement learning, we derive equivalents of standard policy gradient algorithms for training GFlowNets, as well as experimentally explore their various methodological aspects, including baseline training and advantage estimation. Most importantly, our work is the first to derive and successfully apply proximal policy optimization to GFlowNets, showing its improved convergence speed and data efficiency compared to standard GFlowNet training objectives on benchmarks ranging from synthetic energies to molecular graph generation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 48 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

31 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024GFlowNet Training by Policy Gradients12 Aug

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  9. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  10. 2021GFlowNet Foundationscited by 290
  11. 2019A Theory of Regularized Markov Decision Processescited by 286
  12. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
18 more
  1. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  2. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  3. 2017A unified view of entropy-regularized Markov decision processescited by 182
  4. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  5. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 132
  6. 2022GFlowNets and variational inferencecited by 106
  7. 2022Multi-Objective GFlowNetscited by 101
  8. 2023A theory of continuous generative flow networkscited by 101
  9. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 94
  10. 2023Towards Understanding and Improving GFlowNet Trainingcited by 75
  11. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  12. 2024Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Samplingcited by 56
  13. 2024Improved off-policy training of diffusion samplerscited by 48
  14. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 44
  15. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  16. 2024Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximizationcited by 14
  17. 2025gfnx: Fast and Scalable Library for Generative Flow Networks in JAXcited by 3
  18. 2026Evaluating GFlowNet from partial episodes for stable and flexible policy-based trainingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MIPS: Mitigating Mode Collapse in Reinforcement Learning for Large Outcome Spacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Random Policy Evaluation Uncovers Policies of Generative Flow Networks
  2. 2024Improving GFlowNets with Monte Carlo Tree Search
  3. 2024Enhancing Solution Efficiency in Reinforcement Learning: Leveraging Sub-GFlowNet and Entropy Integration
  4. 2024QGFN: Controllable Greediness with Action Values
  5. 2026Towards Understanding the Training of GFlowNets: A Theoretical Perspective
  6. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling
  7. 2025From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster training
  8. 2024Action abstractions for amortized sampling