mlsanityData to 7 Oct 2026

MA-VLCM: A Vision Language Critic Model for Value Estimation of Policies in Multi-Agent Team Settings

arXiv 2603.15418 ↗First posted 16 Mar 2026cs.RO cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Multi-agent reinforcement learning (MARL) commonly relies on a centralized critic to estimate the value function. However, learning such a critic from scratch is highly sample-inefficient and often lacks generalization across environments. At the same time, large vision-language-action models (VLAs) trained on internet-scale data exhibit strong multimodal reasoning and zero-shot generalization capabilities, yet directly deploying them for robotic execution remains computationally prohibitive, particularly in heterogeneous multi-robot systems with diverse embodiments and resource constraints. To address these challenges, we propose Multi-Agent Vision-Language-Critic Models (MA-VLCM), a framework that replaces the learned centralized critic in MARL with a pretrained vision-language model fine-tuned to evaluate multi-agent behavior. MA-VLCM acts as a centralized critic conditioned on natural language task descriptions, visual trajectory observations, and structured multi-agent state information. By eliminating critic learning during policy optimization, our approach significantly improves sample efficiency while producing compact execution policies suitable for deployment on resource-constrained robots. Results show good zero-shot return estimation on models with differing VLM backbones on in-distribution and out-of-distribution scenarios in multi-agent team settings

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 26 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

25 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KL4 Feb

Other references · 24

  1. 2017Graph Attention Networkscited by 8,854
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  8. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  9. 2021How Attentive are Graph Attention Networks?cited by 818
  10. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  11. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  12. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 207
12 more
  1. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  2. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  3. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  4. 2019A Review of Cooperative Multi-Agent Deep Reinforcement Learningcited by 147
  5. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  6. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  7. 2022TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Modelscited by 48
  8. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  9. 2022Contrastive Identity-Aware Learning for Multi-Agent Value Decompositioncited by 19
  10. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Roboticscited by 17
  11. 2024Representation Learning For Efficient Deep Multi-Agent Reinforcement Learningcited by 3
  12. 2020Resilient Monitoring in Heterogeneous Multi-robot Systems through Network Reconfigurationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning
  2. 2026Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
  3. 2026Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning
  4. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success
  5. 2024From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Models
  6. 2026Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization
  7. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  8. 2026Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models