mlsanityData to 7 Oct 2026

Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation

arXiv 2403.08282 ↗First posted 13 Mar 2024cs.CVcited by 24

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Due to the dynamic and unpredictable open-world setting, navigating complex environments in Minecraft poses significant challenges for multi-agent systems. Agents must interact with the environment and coordinate their actions with other agents to achieve common objectives. However, traditional approaches often struggle to efficiently manage inter-agent communication and task distribution, crucial for effective multi-agent navigation. Furthermore, processing and integrating multi-modal information (such as visual, textual, and auditory data) is essential for agents to comprehend their goals and navigate the environment successfully and fully. To address this issue, we design the HAS framework to auto-organize groups of LLM-based agents to complete navigation tasks. In our approach, we devise a hierarchical auto-organizing navigation system, which is characterized by 1) a hierarchical system for multi-agent organization, ensuring centralized planning and decentralized execution; 2) an auto-organizing and intra-communication mechanism, enabling dynamic group adjustment under subtasks; 3) a multi-modal information platform, facilitating multi-modal perception to perform the three navigation tasks with one system. To assess organizational behavior, we design a series of navigation tasks in the Minecraft environment, which includes searching and exploring. We aim to develop embodied organizations that push the boundaries of embodied AI, moving it towards a more human-like organizational structure.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 62 of its 77 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

62 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach19 Dec
  2. 2023See and Think: Embodied Agent in Virtual Environment26 Nov
  3. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory25 May
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  5. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 57

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
38 more
  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  9. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  11. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  12. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  13. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  14. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  15. 2023Large Language Models as Tool Makerscited by 293
  16. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  17. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  18. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  19. 2023Playing repeated games with Large Language Modelscited by 213
  20. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  21. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  22. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  23. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  24. 2019Learning Exploration Policies for Navigationcited by 113
  25. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 105
  26. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  27. 2019Multi-Target Embodied Question Answeringcited by 74
  28. 2023Experiential Co-Learning of Software-Developing Agentscited by 57
  29. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  30. 2023Renderable Neural Radiance Map for Visual Navigationcited by 51
  31. 2021VTNet: Visual Transformer Network for Object Goal Navigationcited by 50
  32. 2022Sound Adversarial Audio-Visual Navigationcited by 46
  33. 2020See, Hear, Explore: Curiosity via Audio-Visual Associationcited by 43
  34. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  35. 2023ChatLLM Network: More brains, More intelligencecited by 37
  36. 2018End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learningcited by 35
  37. 2022Episodic Memory Question Answeringcited by 32
  38. 2021Towards Distraction-Robust Active Visual Trackingcited by 30

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 7; 2026: 4.

24 papers.

Sort:

By area:

  1. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  2. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  3. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  4. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 81
  5. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  6. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  7. 2025Adaptive Graph Pruning for Multi-Agent Communicationcited by 28
  8. 2024CityCraft: A Real Crafter for 3D City Generationcited by 25
  9. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  10. 2024MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needscited by 16
  11. 2024Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaborationcited by 14
  12. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
12 more
  1. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  2. 2025AeroDuo: Aerial Duo for UAV-based Vision and Language Navigationcited by 13
  3. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasoundcited by 8
  4. 2024CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversationscited by 7
  5. 2024Ego3DT: Tracking Every 3D Object in Ego-centric Videoscited by 7
  6. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
  7. 2026Experience Transfer for Multimodal LLM Agents in Minecraft Gamecited by 4
  8. 2025RIG: Synergizing Reasoning and Imagination in End-to-End Generalist Policycited by 3
  9. 2025From Virtual Agents to Robot Teams: A Multi-Robot Framework Evaluation in High-Stakes Healthcare Contextcited by 3
  10. 2026Gated Coordination for Efficient Multi-Agent Collaboration in Minecraft Gamecited by 3
  11. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 3
  12. 2026DC-Ada: Reward-Only Decentralized Sensor Adaptation for Heterogeneous Multi-Robot Teamscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
  2. 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft
  3. 2024S-Agents: Self-organizing Agents in Open-ended Environments
  4. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft
  5. 2024Odyssey: Empowering Minecraft Agents with Open-World Skills
  6. 2025CausalMACE: Causality Empowered Multi-Agents in Minecraft Cooperative Tasks
  7. 2025PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments
  8. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models