mlsanityData to 7 Oct 2026

Thompson sampling for improved exploration in GFlowNets

arXiv 2306.17693 ↗First posted 30 Jun 2023cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Generative flow networks (GFlowNets) are amortized variational inference algorithms that treat sampling from a distribution over compositional objects as a sequential decision-making problem with a learnable action policy. Unlike other algorithms for hierarchical sampling that optimize a variational bound, GFlowNet algorithms can stably run off-policy, which can be advantageous for discovering modes of the target distribution. Despite this flexibility in the choice of behaviour policy, the optimal way of efficiently selecting trajectories for training has not yet been systematically explored. In this paper, we view the choice of trajectories for training as an active learning problem and approach it using Bayesian techniques inspired by methods for multi-armed bandits. The proposed algorithm, Thompson sampling GFlowNets (TS-GFN), maintains an approximate posterior distribution over policies and samples trajectories from this posterior for training. We show in two domains that TS-GFN yields improved exploration and thus faster convergence to the target distribution than the off-policy exploration strategies used in past work.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

37 of 49 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  10. 2016Deep Exploration via Bootstrapped DQNcited by 518
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  12. 2017A Tutorial on Thompson Samplingcited by 420
25 more
  1. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  2. 2021GFlowNet Foundationscited by 286
  3. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  4. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  5. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  6. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  7. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  8. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  9. 2022Biological Sequence Design with GFlowNetscited by 172
  10. 2017Deep Exploration via Randomized Value Functionscited by 155
  11. 2022Bayesian Structure Learning with Generative Flow Networkscited by 153
  12. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  13. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 129
  14. 2022Generative Flow Networks for Discrete Probabilistic Modelingcited by 113
  15. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  16. 2022GFlowNets and variational inferencecited by 103
  17. 2022Multi-Objective GFlowNetscited by 100
  18. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  19. 2023Better Training of GFlowNets with Local Credit and Incomplete Trajectoriescited by 83
  20. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  21. 2023Robust Scheduling with GFlowNetscited by 67
  22. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  23. 2022Generative Augmented Flow Networkscited by 48
  24. 2023GFlowNet-EM for learning compositional latent variable modelscited by 40
  25. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 15; 2025: 5; 2026: 5.

31 papers.

Sort:

By area:

  1. 2023Generative Flow Networks as Entropy-Regularized RLcited by 60
  2. 2024Ant Colony Sampling with GFlowNets for Combinatorial Optimizationcited by 53
  3. 2024Improved off-policy training of diffusion samplerscited by 47
  4. 2023Local Search GFlowNetscited by 45
  5. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 40
  6. 2024Genetic-guided GFlowNets for Sample Efficient Molecular Optimizationcited by 36
  7. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  8. 2023Learning to Scale Logits for Temperature-Conditional GFlowNetscited by 26
  9. 2024QGFN: Controllable Greediness with Action Valuescited by 18
  10. 2024Sampling from Energy-based Policies using Diffusioncited by 15
  11. 2023Order-Preserving GFlowNetscited by 13
  12. 2024Pessimistic Backward Policy for GFlowNetscited by 13
19 more
  1. 2024Improved Off-policy Reinforcement Learning in Biological Sequence Designcited by 13
  2. 2024Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networkscited by 9
  3. 2023Expected flow networks in stochastic environments and two-player zero-sum gamescited by 7
  4. 2023DGFN: Double Generative Flow Networkscited by 7
  5. 2024Investigating Generalization Behaviours of Generative Flow Networkscited by 6
  6. 2024MetaGFN: Exploring Distant Modes with Adapted Metadynamics for Continuous GFlowNetscited by 6
  7. 2025Adaptive Destruction Processes for Diffusion Samplerscited by 6
  8. 2024GFlowNet Training by Policy Gradientscited by 4
  9. 2025Loss-Guided Auxiliary Agents for Overcoming Mode Collapse in GFlowNetscited by 4
  10. 2024Evolution Guided Generative Flow Networkscited by 3
  11. 2026Avoid What You Know: Divergent Trajectory Balance for GFlowNetscited by 3
  12. 2026Discrete diffusion samplers and bridges: Off-policy algorithms and applications in latent spacescited by 2
  13. 2025Improved Exploration in GFlownets via Enhanced Epistemic Neural Networkscited by 1
  14. 2026Evaluating GFlowNet from partial episodes for stable and flexible policy-based trainingcited by 1
  15. 2024Dynamic Backtracking in GFlowNets: Enhancing Decision Steps with Reward-Dependent Adjustment Mechanismscited by 0
  16. 2024Enhancing Solution Efficiency in Reinforcement Learning: Leveraging Sub-GFlowNet and Entropy Integrationcited by 0
  17. 2025Multi-Armed Sampling Problem and the End of Explorationcited by 0
  18. 2026Signal from Structure: Exploiting Submodular Upper Bounds in Generative Flow Networkscited by 0
  19. 2026Exploring Multiple High-Scoring Subspaces in Generative Flow Networkscited by 0