mlsanityData to 14 Oct 2026

AVA: Attentive VLM Agent for Mastering StarCraft II

arXiv 2503.05383 ↗First posted 7 Mar 2025cs.AI cs.MAcited by 5

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

We introduce AVACraft, a multimodal StarCraft II benchmark supporting both Multi-Agent Reinforcement Learning (MARL) and Vision-Language Model (VLM) paradigms. Unlike SMAC-family environments that rely on abstract state representations and exclude VLMs, AVACraft provides RGB visuals, natural language observations, and structured state information, enabling systematic comparison between training-based and zero-shot methods across 21 scenarios spanning micromanagement, coordination, and strategic planning. We establish comprehensive baselines: six MARL algorithms (IQL, QMIX, QTRAN, VDN, MAPPO, IPPO) with Swin-Transformer backbones trained for 5M steps, and multiple VLMs including proprietary (GPT-4o) and open-source (Qwen3-VL) models. Results reveal complementary strengths-MARL peaks at 19.3% win rate after 5M steps, while VLMs achieve 75-90% zero-shot with human-aligned decisions-exposing trade-offs between training efficiency, performance ceilings, interpretability, and deployment cost. Code: https://github.com/camel-ai/VLM-Play-StarCraft2.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 45 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

26 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games20 Nov
  2. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  7. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  8. 2023Large Language Modelscited by 780
  9. 2019The StarCraft Multi-Agent Challengecited by 597
  10. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  11. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  12. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
11 more
  1. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approachcited by 79
  2. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  3. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 68
  4. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  5. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  6. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 34
  7. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Modelscited by 29
  8. 2020TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Gamecited by 21
  9. 2024LLM-PySC2: Starcraft II learning environment for Large Language Modelscited by 9
  10. 2024Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignmentcited by 8
  11. 2024SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMACcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agentscited by 4
  2. 2025EvoCurr: Self-evolving Curriculum with Behavior Code Generation for Complex Decision-makingcited by 3
  3. 2025Subgoal Graph-Augmented Planning for LLM-Guided Open-World Reinforcement Learningcited by 1
  4. 2026EgoEsportsQA: An Egocentric Video Benchmark for Perception and Reasoning in Esportscited by 1
  5. 2026STRATA: Self-Learning Through Role-Aligned Tiered Agents for Real-Time Strategy Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026World Models for Policy Refinement in StarCraft II
  2. 2023AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning
  3. 2025Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning
  4. 2026OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics
  5. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play
  6. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning
  7. 2025HLSMAC: A New StarCraft Multi-Agent Challenge for High-Level Strategic Decision-Making
  8. 2025SC2Arena and StarEvolve: Benchmark and Self-Improvement Framework for LLMs in Complex Decision-Making Tasks