# GFlowNets with Human Feedback

arXiv: https://arxiv.org/abs/2305.07036 · HTML: https://mlsanity.com/paper/2305.07036/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 17 of its 18 references that are in the map.
- First posted on arXiv 11 May 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [Bayesian Structure Learning with Generative Flow Networks](https://mlsanity.com/paper/2202.13903/) (2022)
- [Generative Flow Networks for Discrete Probabilistic Modeling](https://mlsanity.com/paper/2202.01361/) (2022)
- [DAG Matters! GFlowNets Enhanced Explainer For Graph Neural Networks](https://mlsanity.com/paper/2303.02448/) (2023)
- [CFlowNets: Continuous Control with Generative Flow Networks](https://mlsanity.com/paper/2303.02430/) (2023)
- [GFlowCausal: Generative Flow Networks for Causal Discovery](https://mlsanity.com/paper/2210.08185/) (2022)
- [Generative Flow Networks for Precise Reward-Oriented Active Learning on Graphs](https://mlsanity.com/paper/2304.11989/) (2023)

## Built on by (most cited first, 3 of 3)

- [Expert-Aided Causal Discovery of Ancestral Graphs](https://mlsanity.com/paper/2309.12032/) (2023)
- [Random Policy Evaluation Uncovers Policies of Generative Flow Networks](https://mlsanity.com/paper/2406.02213/) (2024)
- [Bifurcated Generative Flow Networks](https://mlsanity.com/paper/2406.01901/) (2024)

## Abstract (arXiv)

We propose the GFlowNets with Human Feedback (GFlowHF) framework to improve the exploration ability when training AI models. For tasks where the reward is unknown, we fit the reward function through human evaluations on different trajectories. The goal of GFlowHF is to learn a policy that is strictly proportional to human ratings, instead of only focusing on human favorite ratings like RLHF. Experiments show that GFlowHF can achieve better exploration ability than RLHF.