# GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networks

arXiv: https://arxiv.org/abs/2503.06514 · HTML: https://mlsanity.com/paper/2503.06514/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 9 Mar 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning](https://mlsanity.com/paper/2405.10292/) (2024)
- [GFlowNets and variational inference](https://mlsanity.com/paper/2210.00580/) (2022)
- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [Robust Scheduling with GFlowNets](https://mlsanity.com/paper/2302.05446/) (2023)
- [Ant Colony Sampling with GFlowNets for Combinatorial Optimization](https://mlsanity.com/paper/2403.07041/) (2024)
- [Vision-Language Models Provide Promptable Representations for Reinforcement Learning](https://mlsanity.com/paper/2402.02651/) (2024)
- [Let the Flows Tell: Solving Graph Combinatorial Optimization Problems with GFlowNets](https://mlsanity.com/paper/2305.17010/) (2023)
- [Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples](https://mlsanity.com/paper/2406.05673/) (2024)
- [DAG Matters! GFlowNets Enhanced Explainer For Graph Neural Networks](https://mlsanity.com/paper/2303.02448/) (2023)
- [Improved Off-policy Reinforcement Learning in Biological Sequence Design](https://mlsanity.com/paper/2410.04461/) (2024)
- [Generalized Mission Planning for Heterogeneous Multi-Robot Teams via LLM-constructed Hierarchical Trees](https://mlsanity.com/paper/2501.16539/) (2025)
- [GFlowNet Fine-tuning for Diverse Correct Solutions in Mathematical Reasoning Tasks](https://mlsanity.com/paper/2410.20147/) (2024)
- [TacoGFN: Target-conditioned GFlowNet for Structure-based Drug Design](https://mlsanity.com/paper/2310.03223/) (2023)

## Built on by (most cited first, 15 of 15)

- [Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.21277/) (2025)
- [LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning](https://mlsanity.com/paper/2510.04573/) (2025)
- [Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need?](https://mlsanity.com/paper/2510.16582/) (2025)
- [RoboAgent: Chaining Basic Capabilities for Embodied Task Planning](https://mlsanity.com/paper/2604.07774/) (2026)
- [AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents](https://mlsanity.com/paper/2605.17933/) (2026)
- [Graph-Grounded LLMs: Leveraging Graphical Function Calling to Minimize LLM Hallucinations](https://mlsanity.com/paper/2503.10941/) (2025)
- [GFlowGR: Fine-tuning Generative Recommendation Frameworks with Generative Flow Networks](https://mlsanity.com/paper/2506.16114/) (2025)
- [Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization](https://mlsanity.com/paper/2601.06224/) (2026)
- [KGLAMP: Knowledge Graph-guided Language model for Adaptive Multi-robot Planning and Replanning](https://mlsanity.com/paper/2602.04129/) (2026)
- [DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum](https://mlsanity.com/paper/2609.19801/) (2026)
- [Iterative Foundation Model Fine-Tuning on Multiple Rewards](https://mlsanity.com/paper/2511.00220/) (2025)
- [Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams](https://mlsanity.com/paper/2603.08814/) (2026)
- [AgentSpec: Understanding Embodied Agent Scaffolds Through Controlled Composition](https://mlsanity.com/paper/2606.14674/) (2026)
- [Interpretable epistemic uncertainty decomposition in sequential generative models via polynomial chaos surrogates](https://mlsanity.com/paper/2510.21523/) (2025)
- [Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning](https://mlsanity.com/paper/2609.19878/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) have recently shown promising advancements in sequential decision-making tasks through task-specific fine-tuning. However, common fine-tuning methods, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) techniques like Proximal Policy Optimization (PPO), present notable limitations: SFT assumes Independent and Identically Distributed (IID) data, while PPO focuses on maximizing cumulative rewards. These limitations often restrict solution diversity and hinder generalization in multi-step reasoning tasks. To address these challenges, we introduce a novel framework, GFlowVLM, a framework that fine-tune VLMs using Generative Flow Networks (GFlowNets) to promote generation of diverse solutions for complex reasoning tasks. GFlowVLM models the environment as a non-Markovian decision process, allowing it to capture long-term dependencies essential for real-world applications. It takes observations and task descriptions as inputs to prompt chain-of-thought (CoT) reasoning which subsequently guides action selection. We use task based rewards to fine-tune VLM with GFlowNets. This approach enables VLMs to outperform prior fine-tuning methods, including SFT and RL. Empirical results demonstrate the effectiveness of GFlowVLM on complex tasks such as card games (NumberLine, BlackJack) and embodied planning tasks (ALFWorld), showing enhanced training efficiency, solution diversity, and stronger generalization capabilities across both in-distribution and out-of-distribution scenarios.