mlsanityData to 7 Oct 2026

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

arXiv 2405.10292 ↗First posted 16 May 2024cs.AI cs.CL cs.CVcited by 158

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Large vision-language models (VLMs) fine-tuned on specialized visual instruction-following data have exhibited impressive language reasoning capabilities across various scenarios. However, this fine-tuning paradigm may not be able to efficiently learn optimal decision-making agents in multi-step goal-directed tasks from interactive environments. To address this challenge, we propose an algorithmic framework that fine-tunes VLMs with reinforcement learning (RL). Specifically, our framework provides a task description and then prompts the VLM to generate chain-of-thought (CoT) reasoning, enabling the VLM to efficiently explore intermediate reasoning steps that lead to the final text-based action. Next, the open-ended text output is parsed into an executable action to interact with the environment to obtain goal-directed task rewards. Finally, our framework uses these task rewards to fine-tune the entire VLM with RL. Empirically, we demonstrate that our proposed framework enhances the decision-making capabilities of VLM agents across various tasks, enabling 7b models to outperform commercial models such as GPT4-V or Gemini. Furthermore, we find that CoT reasoning is a crucial component for performance improvement, as removing the CoT reasoning results in a significant decrease in the overall performance of our method.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 65 of its 80 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

65 of 80 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
38 more
  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023Mistral 7Bcited by 3,065
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2015Trust Region Policy Optimizationcited by 2,503
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  13. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  14. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  15. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  16. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  17. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  18. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  19. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  20. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  21. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  22. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  23. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  24. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  25. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  26. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  27. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  28. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  29. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  30. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  31. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  32. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  33. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  34. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  35. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  36. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  37. 2020The NetHack Learning Environmentcited by 196
  38. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 89; 2026: 44.

158 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  4. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  5. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  6. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  7. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  8. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  9. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  10. 2025Agentic Reinforced Policy Optimizationcited by 166
  11. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  12. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
38 more
  1. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  2. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  3. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 129
  4. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  5. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  6. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  7. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  8. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  9. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  10. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  11. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 52
  12. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  13. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  14. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  15. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  16. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
  17. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  18. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  19. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  20. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  21. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  22. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 27
  23. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  24. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
  25. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  26. 2024Steering Large Language Models between Code Execution and Textual Reasoningcited by 24
  27. 2024Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Casecited by 23
  28. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 23
  29. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 23
  30. 2025Digi-Q: Learning Q-Value Functions for Training Device-Control Agentscited by 20
  31. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  32. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 19
  33. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agentscited by 19
  34. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  35. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  36. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  37. 2025Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learningcited by 17
  38. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17

50 most cited of 158 All 158 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning
  2. 2025Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  4. 2024Improve Vision Language Model Chain-of-thought Reasoning
  5. 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme
  6. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models
  7. 2025Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
  8. 2025The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs