mlsanityData to 7 Oct 2026

Language Agents as Optimizable Graphs

arXiv 2402.16823 ↗First posted 26 Feb 2024cs.AI cs.CL cs.LGcited by 81

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Various human-designed prompt engineering techniques have been proposed to improve problem solvers based on Large Language Models (LLMs), yielding many disparate code bases. We unify these approaches by describing LLM-based agents as computational graphs. The nodes implement functions to process multimodal data or query LLMs, and the edges describe the information flow between operations. Graphs can be recursively combined into larger composite graphs representing hierarchies of inter-agent collaboration (where edges connect operations of different agents). Our novel automatic graph optimizers (1) refine node-level LLM prompts (node optimization) and (2) improve agent orchestration by changing graph connectivity (edge optimization). Experiments demonstrate that our framework can be used to efficiently develop, integrate, and automatically improve various LLM agents. The code can be found at https://github.com/metauto-ai/gptswarm.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 39 of its 55 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

39 of 55 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct
  2. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration3 Oct
  3. 2023AutoAgents: A Framework for Automatic Agent Generation29 Sep
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  5. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  6. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
20 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  3. 2023Large Language Models as Optimizerscited by 830
  4. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  5. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  6. 2020Aligning AI With Shared Human Valuescited by 706
  7. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  8. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  11. 2021D'ya like DAGs? A Survey on Structure Learning and Causal Discoverycited by 209
  12. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  13. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 105
  14. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  15. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  16. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  17. 2020Meta Learning Backpropagation And Improving Itcited by 49
  18. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
  19. 2022Eliminating Meta Optimization Through Self-Referential Meta Learningcited by 12
  20. 2013Hierarchical self-organization of non-cooperating individualscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 23; 2026: 47.

81 papers.

Sort:

By area:

  1. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  2. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  3. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  7. 2024Symbolic Learning Enables Self-Evolving Agentscited by 104
  8. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  9. 2026Self-Harness: Harnesses That Improve Themselvescited by 79
  10. 2025Latent Collaboration in Multi-Agent Systemscited by 67
  11. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  12. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
38 more
  1. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 46
  2. 2024Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agentscited by 35
  3. 2025Flow: Modularized Agentic Workflow Automationcited by 31
  4. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 30
  5. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  6. 2025MAS-ZERO: Designing Multi-Agent Systems with Zero Supervisioncited by 26
  7. 2025OptimAI: Optimization from Natural Language Using LLM-Powered AI Agentscited by 22
  8. 2024REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimizationcited by 20
  9. 2026CooperBench: Why Coding Agents Cannot be Your Teammates Yetcited by 20
  10. 2025Graph World Modelcited by 14
  11. 2025Dynamic Generation of Multi-LLM Agents Communication Topologies with Graph Diffusion Modelscited by 11
  12. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  13. 2026GraphPlanner: Graph Memory-Augmented Agentic Routing for Multi-Agent LLMscited by 10
  14. 2026WHALE: A Simple Recipe for Joint Harness-Weight Optimizationcited by 9
  15. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  16. 2026The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systemscited by 8
  17. 2026TeamBench: Evaluating Agent Coordination under Enforced Role Separationcited by 8
  18. 2026AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systemscited by 8
  19. 2026AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentationcited by 7
  20. 2025Toward a Safe Internet of Agentscited by 6
  21. 2025SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoningcited by 5
  22. 2026MonoScale: Scaling Multi-Agent System with Monotonic Improvementcited by 5
  23. 2026Mitigating Conversational Inertia in Multi-Turn Agentscited by 5
  24. 2026Harnessing the Collective Intelligence of AI Agents in the Wild for New Discoveriescited by 5
  25. 2025The Society of HiveMind: Multi-Agent Optimization of Foundation Model Swarms to Unlock the Potential of Collective Intelligencecited by 4
  26. 2026El Agente Gr'afico: A Semantic Execution Runtime for Scientific Agentscited by 4
  27. 2024Memory-Augmented Agent Training for Business Document Understandingcited by 3
  28. 2025Agent WARPP: Workflow Adherence via Runtime Parallel Personalizationcited by 3
  29. 2025Reducing Cognitive Overhead in Tool Use via Multi-Small-Agent Reinforcement Learningcited by 3
  30. 2025Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimizationcited by 3
  31. 2025LoopBench: Discovering Emergent Symmetry Breaking Strategies with LLM Swarmscited by 3
  32. 2026Architecting Trust in Artificial Epistemic Agentscited by 3
  33. 2026Swarm Skills: A Portable, Self-Evolving Multi-Agent System Specification for Coordination Engineeringcited by 3
  34. 2025JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluationcited by 2
  35. 2026JudgeFlow: Agentic Workflow Optimization via Block Judgecited by 2
  36. 2026Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learningcited by 2
  37. 2026Learning to Interrupt in Language-based Multi-agent Communicationcited by 2
  38. 2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflowscited by 2

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration
  2. 2025Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects
  3. 2025A Survey on the Optimization of Large Language Model-based Agents
  4. 2025AMAS: Adaptively Determining Communication Topology for LLM-based Multi-Agent System
  5. 2024Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
  6. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges
  7. 2024Input Conditioned Graph Generation for Language Agents
  8. 2025Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development