mlsanityData to 7 Oct 2026

Building Cooperative Embodied Agents Modularly with Large Language Models

arXiv 2307.02485 ↗First posted 5 Jul 2023cs.AI cs.CL cs.CVcited by 231

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments. While previous research either presupposes a cost-free communication channel or relies on a centralized controller with shared observations, we harness the commonsense knowledge, reasoning ability, language comprehension, and text generation prowess of LLMs and seamlessly incorporate them into a cognitive-inspired modular framework that integrates with perception, memory, and execution. Thus building a Cooperative Embodied Language Agent CoELA, who can plan, communicate, and cooperate with others to accomplish long-horizon tasks efficiently. Our experiments on C-WAH and TDW-MAT demonstrate that CoELA driven by GPT-4 can surpass strong planning-based methods and exhibit emergent effective communication. Though current Open LMs like LLAMA-2 still underperform, we fine-tune a CoELA with data collected with our agents and show how they can achieve promising performance. We also conducted a user study for human-agent interaction and discovered that CoELA communicating in natural language can earn more trust and cooperate more effectively with humans. Our research underscores the potential of LLMs for future research in multi-agent cooperation. Videos can be found on the project website https://vis-www.cs.umass.edu/Co-LLM-Agents/.

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 57 of its 87 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

57 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents3 Feb

Other references · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2017Mask R-CNNcited by 7,259
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  11. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
38 more
  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2019The StarCraft Multi-Agent Challengecited by 589
  7. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  11. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  13. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  14. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  15. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  16. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  17. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  18. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  19. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  20. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  21. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  22. 2018Learning Attentional Communication for Multi-Agent Cooperationcited by 170
  23. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  24. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  25. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  26. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 131
  27. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  28. 2021Skill Induction and Planning with Latent Languagecited by 99
  29. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  30. 2021ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mindcited by 57
  31. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  32. 2022Retrospectives on the Embodied AI Workshopcited by 46
  33. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  34. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  35. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  36. 2019Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agentscited by 42
  37. 2022Plansformer: Generating Symbolic Plans using Transformerscited by 41
  38. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 84; 2025: 83; 2026: 42.

231 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  5. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  6. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  7. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  10. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  11. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  12. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 163
38 more
  1. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  3. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  6. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  7. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  8. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  9. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  10. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  11. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  12. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  13. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
  14. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  15. 2023MindAgent: Emergent Gaming Interactioncited by 87
  16. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  17. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 81
  18. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  19. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  20. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Modelscited by 65
  21. 2024ProtAgents: Protein discovery via large language model multi-agent collaborations combining physics and machine learningcited by 63
  22. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  23. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  24. 2023Multi-Agent Consensus Seeking via Large Language Modelscited by 50
  25. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  26. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  27. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 46
  28. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  29. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 45
  30. 2024Enhance Reasoning for Large Language Models in the Game Werewolfcited by 44
  31. 2023Digital Life Project: Autonomous 3D Characters with Social Intelligencecited by 43
  32. 2024COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Modelscited by 43
  33. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agentscited by 41
  34. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  35. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
  36. 2026Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoningcited by 40
  37. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  38. 2024Exploring Prosocial Irrationality for LLM Agents: A Social Cognition Viewcited by 38

50 most cited of 231 All 231 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning
  2. 2026LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behavior
  3. 2026Communication-Efficient Digital-Twin Coordination for Heterogeneous LLM Embodied Agents over Computing Power Networks
  4. 2026Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue
  5. 2025Multi-Agent Language Models: Advancing Cooperation, Coordination, and Adaptation
  6. 2025Learn as Individuals, Evolve as a Team: Multi-agent LLMs Adaptation in Embodied Environments
  7. 2026COOP2: Defining, Observing, and Repairing Cooperation in LLM Multi-Agent Systems
  8. 2025Why do AI agents communicate in human language?