mlsanityData to 14 Oct 2026

Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Control

arXiv 2412.11761 ↗First posted 16 Dec 2024cs.AIcited by 4

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance across various tasks. Their potential to facilitate human coordination with many agents is a promising but largely under-explored area. Such capabilities would be helpful in disaster response, urban planning, and real-time strategy scenarios. In this work, we introduce (1) a real-time strategy game benchmark designed to evaluate these abilities and (2) a novel framework we term HIVE. HIVE empowers a single human to coordinate swarms of up to 2,000 agents through a natural language dialog with an LLM. We present promising results on this multi-agent benchmark, with our hybrid approach solving tasks such as coordinating agent movements, exploiting unit weaknesses, leveraging human annotations, and understanding terrain and strategic points. Our findings also highlight critical limitations of current models, including difficulties in processing spatial visual information and challenges in formulating long-term strategic plans. This work sheds light on the potential and limitations of LLMs in human-swarm coordination, paving the way for future research in this area. The HIVE project page, hive.syrkis.com, includes videos of the system in action.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 27 of its 74 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

27 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration18 Apr
  2. 2024LLM Multi-Agent Systems: Challenges and Open Problems5 Feb
  3. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges21 Jan

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  4. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  5. 2019The StarCraft Multi-Agent Challengecited by 597
  6. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  7. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  8. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  9. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
  10. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  11. 2017Behavior Trees in Robotics and AI: An Introductioncited by 107
  12. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 99
12 more
  1. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 66
  2. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  3. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  4. 2023Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framingcited by 51
  5. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 48
  6. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  7. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  8. 2023Robot Behavior-Tree-Based Task Generation with Large Language Modelscited by 23
  9. 2024SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language modelscited by 19
  10. 2023Investigating Agency of LLMs in Human-AI Collaboration Taskscited by 15
  11. 2024A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environmentcited by 13
  12. 2024CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  2. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 2
  3. 2025Tool-RoCo: An Agent-as-Tool Self-organization Large Language Model Benchmark in Multi-robot Cooperationcited by 2
  4. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking LLMs' Swarm intelligence
  2. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models
  3. 2025LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems
  4. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
  5. 2026SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?
  6. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
  7. 2024BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systems
  8. 2025Can LLM Agents Solve Collaborative Tasks? A Study on Urgency-Aware Planning and Coordination