# Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization

arXiv: https://arxiv.org/abs/2410.15474 · HTML: https://mlsanity.com/paper/2410.15474/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 52 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [GFlowNet Foundations](https://mlsanity.com/paper/2111.09266/) (2021)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [Trajectory balance: Improved credit assignment in GFlowNets](https://mlsanity.com/paper/2201.13259/) (2022)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [A unified view of entropy-regularized Markov decision processes](https://mlsanity.com/paper/1705.07798/) (2017)
- [Non-stationary Stochastic Optimization](https://mlsanity.com/paper/1307.5449/) (2013)
- [Biological Sequence Design with GFlowNets](https://mlsanity.com/paper/2203.04115/) (2022)
- [Learning GFlowNets from partial episodes for improved convergence and stability](https://mlsanity.com/paper/2209.12782/) (2022)
- [Generative Flow Networks for Discrete Probabilistic Modeling](https://mlsanity.com/paper/2202.01361/) (2022)
- [GFlowNets and variational inference](https://mlsanity.com/paper/2210.00580/) (2022)
- [Multi-Objective GFlowNets](https://mlsanity.com/paper/2210.12765/) (2022)
- [Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review](https://mlsanity.com/paper/2407.13734/) (2024)
- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [Reward is enough for convex MDPs](https://mlsanity.com/paper/2106.00661/) (2021)
- [Towards Understanding and Improving GFlowNet Training](https://mlsanity.com/paper/2305.07170/) (2023)
- [Amortizing intractable inference in diffusion models for vision, language, and control](https://mlsanity.com/paper/2405.20971/) (2024)
- [Munchausen Reinforcement Learning](https://mlsanity.com/paper/2007.14430/) (2020)
- [Generative Flow Networks as Entropy-Regularized RL](https://mlsanity.com/paper/2310.12934/) (2023)
- [Sample-efficient Multi-objective Molecular Optimization with GFlowNets](https://mlsanity.com/paper/2302.04040/) (2023)
- [Local Search GFlowNets](https://mlsanity.com/paper/2310.02710/) (2023)
- [Molecular Mechanics-Driven Graph Neural Network with Multiplex Graph for Molecular Structures](https://mlsanity.com/paper/2011.07457/) (2020)
- [Discrete Probabilistic Inference as Control in Multi-path Environments](https://mlsanity.com/paper/2402.10309/) (2024)
- [Fast Rates for Maximum Entropy Exploration](https://mlsanity.com/paper/2303.08059/) (2023)

## Built on by (most cited first, 14 of 14)

- [Revisiting Non-Acyclic GFlowNets in Discrete Environments](https://mlsanity.com/paper/2502.07735/) (2025)
- [Adaptive Destruction Processes for Diffusion Samplers](https://mlsanity.com/paper/2506.01541/) (2025)
- [gfnx: Fast and Scalable Library for Generative Flow Networks in JAX](https://mlsanity.com/paper/2511.16592/) (2025)
- [Learning Shortest Paths with Generative Flow Networks](https://mlsanity.com/paper/2603.01786/) (2026)
- [Evaluating GFlowNet from partial episodes for stable and flexible policy-based training](https://mlsanity.com/paper/2603.01047/) (2026)
- [Proximal Policy Optimization for Amortized Discrete Sampling](https://mlsanity.com/paper/2606.15793/) (2026)
- [Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Training](https://mlsanity.com/paper/2505.20110/) (2025)
- [Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNets](https://mlsanity.com/paper/2602.10583/) (2026)
- [Your GFlowNet Secretly Learns an Optimal Transport Plan](https://mlsanity.com/paper/2606.06272/) (2026)
- [AlphaG-OPD: Reliability-Gated Sibling Counterfactuals for On-Policy Distillation in Symbolic Alpha Factor Discovery](https://mlsanity.com/paper/2608.01303/) (2026)
- [Bayesian Symbolic Regression with Entropic Reinforcement Learning](https://mlsanity.com/paper/2608.09617/) (2026)
- [Particle GFlowNets: Rethinking Generative Marginalization Models](https://mlsanity.com/paper/2609.11538/) (2026)
- [Learning a Mixture of GFlowNets](https://mlsanity.com/paper/2610.07562/) (2026)
- [Towards Understanding the Training of GFlowNets: A Theoretical Perspective](https://mlsanity.com/paper/2610.13711/) (2026)

## Abstract (arXiv)

Generative Flow Networks (GFlowNets) are a family of generative models that learn to sample objects with probabilities proportional to a given reward function. The key concept behind GFlowNets is the use of two stochastic policies: a forward policy, which incrementally constructs compositional objects, and a backward policy, which sequentially deconstructs them. Recent results show a close relationship between GFlowNet training and entropy-regularized reinforcement learning (RL) problems with a particular reward design. However, this connection applies only in the setting of a fixed backward policy, which might be a significant limitation. As a remedy to this problem, we introduce a simple backward policy optimization algorithm that involves direct maximization of the value function in an entropy-regularized Markov Decision Process (MDP) over intermediate rewards. We provide an extensive experimental evaluation of the proposed approach across various benchmarks in combination with both RL and GFlowNet algorithms and demonstrate its faster convergence and mode discovery in complex environments.